System and method for diagnosing severity of gastric cancer

ABSTRACT

A diagnostic system, method, and a computer-readable storage medium for determining a severity of a gastric condition, such as gastric cancer, in a subject are disclosed. The diagnostic system includes a processor that can obtain various images of a stomach of the subject including wavelength images and generate difference images from the wavelength images. The processor can compare the subject images with reference images representative of different severity levels of gastric cancer, or can input the subject images into a learned model trained using the reference images stored in the database to extract a feature pattern corresponding to a severity of gastric cancer to diagnose the subject as having a particular severity level of gastric cancer.

BACKGROUND

Gastric conditions, including gastric atrophy, gastritis, and gastriccancer, affect millions of people. Gastric cancer is the most commonform of digestive system malignant tumor among Japanese people. Gastriccancer and other gastric conditions can be cured by early detection andtreatment. However, many cases are detected in an advanced state,resulting in a poor prognostic outcome.

Endoscopic examination and imaging offer opportunities for earlydetection. For example, with an endoscopic apparatus, a doctor canobserve the organs in the body cavity and make a diagnosis by insertingthe elongated insertion portion into the body cavity and using asolid-stage imaging element or the like as an imaging means. However,final diagnoses using these medical imaging apparatuses are mainly basedon determinations by doctors, which are inherently subjective. Forexample, variations in experience and knowledge between doctors mayresult in inconsistent diagnoses.

SUMMARY

The present systems and methods provide quantitative and objectivediagnostic support information via image processing to facilitateconsistent and accurate diagnoses of gastric conditions, such as gastriccancer and the severity thereof.

The disclosed embodiments include a diagnostic system for determining aseverity of gastric cancer in a subject, a method for determining aseverity of gastric cancer in a subject, and a computer-readable storagemedium storing a computer-executable program that causes a computer toperform functions for determining a severity of gastric cancer in asubject. The severity of gastric cancer may include determining a stageof gastric cancer.

The diagnostic system according to the disclosed embodiments includes aprocessor programmed to obtain various images of a stomach of thesubject including wavelength images, and generate difference images fromthe wavelength images. The processor is programmed to compare thesubject images with reference images representative of differentseverity levels of gastric cancer, or input the subject images into alearned model trained using the reference images stored in the databaseto extract a feature pattern corresponding to a severity of gastriccancer to diagnose the subject as having a particular severity level ofgastric cancer.

The method for determining a severity of gastric cancer in a subject mayinclude obtaining various images of a stomach of the subject includingwavelength images, and generating difference images from the wavelengthimages. The subject images may then be compared with reference imagesrepresentative of different severity levels of gastric cancer, or inputinto a learned model trained using the reference images stored in thedatabase to extract a feature pattern corresponding to a severity ofgastric cancer to diagnose the subject as having a particular severitylevel of gastric cancer.

A computer-readable storage medium according to the disclosedembodiments stores a computer-executable program that causes a computerto perform functions, such as obtaining various images of a stomach ofthe subject including wavelength images, generating difference imagesfrom the wavelength images, and comparing the subject images withreference images representative of different severity levels of gastriccancer, or inputting the subject images into a learned model trainedusing the reference images stored in the database to extract a featurepattern corresponding to a severity of gastric cancer to diagnose thesubject as having a particular severity level of gastric cancer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary diagnostic processor system.

FIG. 2 is a flow chart of an exemplary diagnosis method.

FIG. 3 is a flow chart of an exemplary method for generating subjectabnormality images.

FIG. 4 is a flow chart of an exemplary method for categorizing referenceand standard data.

FIG. 5 is a flow chart of an exemplary method for generating andanalyzing abnormality data.

FIG. 6 is a flow chart of an exemplary method for diagnosing a subjectbased on a comparison of a subject abnormality images and referenceabnormality images.

FIG. 7 is a flow chart of an exemplary method for generating abnormalityscores for a subject.

FIG. 8 is a flow chart of an exemplary method for generating non-imageabnormality scores for a subject.

FIG. 9 is a flow chart of an exemplary method for generating a visualrepresentation of subject abnormality data based on abnormality scores.

FIG. 10 is a flow chart of an exemplary method for generating subjectabnormality images to facilitate diagnosis of a severity level ofgastric atrophy.

FIGS. 11A-D are exemplary abnormality images of stained stomachs.

FIG. 12 is a flow chart of an exemplary method for categorizingreference and standard data to facilitate diagnosis of a severity levelof gastric atrophy.

FIGS. 13A-D are flow charts of exemplary methods for diagnosing aseverity level of gastric atrophy via stained images.

FIG. 14 is a flow chart of an exemplary method for generating andanalyzing abnormality data to facilitate diagnosis of a severity levelof gastric atrophy.

FIG. 15 is a flow chart of an exemplary method for generatingabnormality scores for a subject to facilitate diagnosis of a severitylevel of gastric atrophy.

FIG. 16 is a flow chart of an exemplary method for diagnosing a severityof gastric atrophy and/or possibility of gastric cancer based on thesubject abnormality score.

FIG. 17 is a flow chart of an exemplary method for generating subjectabnormality images to facilitate diagnosis of a severity level ofgastritis.

FIGS. 18A-D are exemplary abnormality images of stomach walls showingcontrast between the stomach wall and a blood vessel.

FIG. 19 is a flow chart of an exemplary method for categorizingreference and standard data to facilitate diagnosis of a severity levelof gastritis.

FIGS. 20A-D are flow charts of an exemplary method for diagnosing aseverity level of gastritis via blood vessel images.

FIG. 21 is a flow chart of an exemplary method for generating andanalyzing abnormality data to facilitate diagnosis of a severity levelof gastritis.

FIG. 22 is a flow chart of an exemplary method for generatingabnormality scores for a subject to facilitate diagnosis of a severitylevel of gastritis.

FIG. 23 is a flow chart of an exemplary method for diagnosing a severityof gastritis and/or possibility of gastric cancer based on the subjectabnormality score.

FIG. 24 is a schematic image of the layers of the stomach.

FIG. 25 is a flow chart of an exemplary method for categorizingreference and standard data to facilitate diagnosis of a severity leveland/or stage of gastric cancer.

FIGS. 26A and 26B are flow charts of exemplary methods/algorithms fordiagnosing a severity of gastric cancer.

FIGS. 27A-27C show exemplary methods for comparing subject images withreference images to diagnose a severity of gastric cancer.

FIGS. 28A and 28B show exemplary ensemble learning models fordetermining a gastric cancer stage and reliability score.

FIG. 29 shows an exemplary method/algorithm for preparing (e.g.,processing) the reference images and training an artificial intelligence(AI) model using the reference images.

DETAILED DESCRIPTION

It will be apparent to the skilled artisan in the medical field fromthis disclosure that the following descriptions of exemplary embodimentsare provided as examples only and need not limit the broad inventiveprinciples described herein or included in the appended claims.

The present disclosure relates to a diagnostic system, non-transitorycomputer-readable storage medium, and method for determining a severityof a gastric condition of a subject based on image and non-image data.For example, the present disclosure relates to comparing an image of thestomach of the subject with a standard stomach image of a healthystomach to generate a subject abnormality image. The subject abnormalityimage may then be compared with reference abnormality images of stomachsrepresentative of different severity levels of gastric conditions. Thesubject may be then diagnosed based at least in part on the comparisonbetween the subject abnormality image and the reference abnormalityimages as having a particular severity level of a gastric condition. Asdiscussed in more detail below, the system and the associatedcomputer-readable storage medium and method enable consistent andaccurate diagnoses of the severity of gastric conditions.

FIG. 1 shows an exemplary processor system 10 for use in connection withdiagnosing severity levels of gastric conditions. The processor system10 may be a general-purpose computer, such as a personal computer,tablet, or mobile device, a specific-purpose computer or workstation, amainframe computer, or a distributed computing system. The processorsystem 10 is configured to execute various software programs, includingsoftware performing all or part of the processes and algorithmsdisclosed herein. The exemplary processor system 10 includes acontroller or processor 12 that is configured to process data, suchimage and non-image information received as inputs for variousalgorithms and software programs. The processor 12 may include hardware,and the hardware may include at least one of a circuit for processingdigital signals and a circuit for processing analog signals, forexample. The processor may include one or a plurality of circuit devices(e.g., an IC) or one or a plurality of circuit elements (e.g., aresistor, a capacitor) on a circuit board, for example. The processor 12may be a central processing unit (CPU), and/or various types ofprocessors, including a GPU (Graphics Processing Unit) and a DSP(Digital Signal Processor), may be used. The processor may be a hardwarecircuit with an ASIC (Application Specific Integrated Circuit) or anFPGA (Field-Programmable Gate Array). The processor may include anamplification circuit, a filter circuit, or the like for processinganalog signals.

The processor 12 may execute operating system instructions, along withsoftware algorithms, computer-executable instructions, and processingfunctions of the system 10. Such algorithms and computer-executableinstructions may be stored in a computer readable-storage medium, suchas storage 14. “Computer readable-storage medium” as used herein refersto a non-transitory computer readable storage medium. The system 10 mayinclude one or more storage devices 14. The storage 14 may include amemory and/or other storage device. The memory may be, for example,random-access memory (RAM) of a computer. The memory may be asemiconductor memory such as an SRAM and a DRAM. The storage device maybe, for example, a register, a magnetic storage device such as a harddisk device, an optical storage device such as an optical disk device,an internal or external hard drive, a server, a solid-state storagedevice, CD-ROM, DVD, other optical or magnetic disk storage, or otherstorage devices. Computer-executable instructions include, for example,instructions and data which cause the processor system 10 to perform acertain function or group of functions. When the instructions areexecuted by the processor 12, the functions of each unit of the systemand the like are implemented. The instructions may be a set ofinstructions constituting a program or an instruction for causing anoperation on the hardware circuit of the processor.

Data, including subject image data, subject non-image data, and otherdata, such as reference images, reference abnormality images, andstandard images may be stored in a database in the storage 13, such asthe memory or another storage device. Such data may also be provided tothe processor 12 by an input device 16, such as a keyboard, touchscreen,mouse, data acquisition device, network device, or any other suitableinput device. Exemplary data acquisition devices may include an imagingsystem or device, such as an endoscope, a subject monitor, or any othersuitable system or device capable of collecting or receiving dataregarding the subject. Subject data may include image data and/ornon-image data, and may include any of static data, dynamic data, andlongitudinal data. For example, subject images collected by an endoscopemay be provided to the processor to diagnose a severity of a gastriccondition. Data, such as subject, standard, and reference images, aswell as non-image data may be stored in a database or various databasesaccessible by the processor 12. The processor may be configured toimplement a deep learning process described in detail below to generatenormal image data.

The various components of the diagnostic system 10 and the like may beconnected with each other via any types of digital data communicationsuch as a communication network 22. Data may also be provided to theprocessor system 10 through a network device 20, such as a wired orwireless Ethernet card, a wireless network adapter, or any other devicesdesigned to facilitate communication with other devices through anetwork 22. The network 22 may be, for example, a Local Area Network(LAN), Wide Area Network (WAN), and computers and networks which formthe Internet. The system 10 may exchange data and communicate with othersystems through the network 22. Although the system shown in FIG. 1 isshown as being connected to a network, the system may be also beconfigured to work offline.

Results, including diagnoses of a severity of a gastric condition outputby the processor 12 may be stored in accordance with one or morealgorithms in one or more storage devices 14, such as memory, mayundergo additional processing, or may be provided to an operator via anoutput device 18, such as a display and/or a printer. Based on thedisplayed or printed output, an operator may request additional oralternative processing or provide additional or alternative data, forexample, via an input device 16.

FIG. 2 shows an exemplary method for diagnosing a subject as having aparticular severity of a gastric condition based on image abnormalitydata. Gastric conditions may include, for example, gastric atrophy,gastritis, and gastric cancer. As shown in FIG. 2, a subject image 30 ofa stomach of the subject is obtained. The subject image 30 is comparedwith a standard image 32 to generate a subject abnormality image 34,which is indicative of differences between the subject image and thestandard image. The subject image 30 may be an image collected by anendoscope and may be processed to improve contrast and/or extract orenhance features. The standard image is an image of a stomach in ahealthy state. The standard image may be an earlier image of the subjectin a healthy state or may be an image of a stomach of a different personin a healthy state. As discussed in more detail below, the differentperson may be selected based on one or more shared characteristics withthe subject, such as age, race, and sex.

Further with respect to FIG. 2, reference images 36 of stomachsexhibiting different severity levels of various gastric conditions arealso compared to the standard image to generate reference abnormalityimages 38, which are indicative of differences between the referenceimages and the standard image. Reference images 36 may be standardizedimages acquired from a database of reference images for each diagnosedcondition or disorder collected from a particular group of peoplediagnosed with such conditions, as discussed above. The reference images36 may be the actual images, optionally processed to enhance thestructural feature of interest, collected from the people of aparticular group or characteristic category. Alternatively, thereference images 36 may be average images created based on the datacollected from the people of a particular population diagnosed with suchconditions. For example, a representative average stomach image for eachgastric condition may be generated. Additionally, representative averagestomach images corresponding to various severity levels within aparticular gastric condition may also be generated. Thus, multiplerepresentative or average images may be created for each gastriccondition and severity level.

The subject abnormality image 34 is then compared 40 with the referenceabnormality images 38. All of the images may be standardized into one ormore common or similar formats to facilitate analysis and comparison.The subject is diagnosed 44 as having a particular severity level of agastric condition based at least on part on the comparison between thesubject abnormality image the reference abnormality images. Thediagnosis 44 may also be made by taking other data 42 and analysis intoconsideration, including non-image data, such as clinical data,laboratory data, subject history, family history, subject vital signs,results of various tests (e.g., genetic tests), and any other relevantnon-image data. Based on the subject and reference data, numerousreference and subject abnormality data and images may be created. Then,a report 46 of the diagnosis 44 is output to an output device 18 (shownin FIG. 1), such as a display or a printer, or may be output to adatabase for storage, or to a user in a human-readable format.

The various images and data described herein may be stored in one ormore databases to facilitate subsequent data analysis. Moreover, any orall of the foregoing comparisons may be performed either automaticallyby a data processing system, such as system 10, or by a medicalprofessional, such as a doctor, or by some combination thereof, tofacilitate automatic or manual diagnosis of the subject in step 44.

The abnormality images described herein may be generated through anysuitable technique. For example, an abnormality image may be adifference image between two or more images. A subject abnormality imagemay be created by subtracting a standard image from a subject image.Likewise, reference abnormality images may be generated by subtracting astandard image from each of the reference images. The resultingdifference (abnormality) images show the differences between the subjectand reference images and the standard image. By eliminating normalfeatures through subtractive processing, the difference images make iteasier to identify areas of abnormalities or deviations from normal. Forexample, the difference images allow a user to focus on, extract, andenhance the deviations and abnormalities in the subject image that maynot be as apparent from a comparison of the raw subject images with thestandard images. In the absence of differential imaging, it is oftendifficult to identify the extent of deviation in a subject image bysimply comparing the subject image and a normal image, for example, in aside-by-side comparison. The difference images enable a user to clearlydetermine the extent of deviation from the standard images, andaccurately diagnose the severity of a particular gastric condition.Using the systems and methods disclosed herein, diagnoses of theseverity of various gastric conditions may become more consistent andobjective.

Standard images, such as normal reference and normal subject images, maybe generated through deep learning techniques. Deep learning is amachine learning technique using multiple data processing layers torecognize various structures in data sets and accurately classify thedata sets. For instance, a deep learning model may be trained togenerate corresponding normal images from abnormal images, such asreference images indicative of a particular level of severity or anabnormal subject image.

Such a deep learning model may be, for example, an autoencoder network.An autoencoder network has at least three layers: an input layer, ahidden layer for encoding, and an output decoding layer. The autoencodernetwork may be a variational autoencoder (VAE) model. The VAE is a classof deep generative models that includes an encoder and a decoder.

The model may be trained based on normal training images only. Suchnormal training images may be images of healthy stomachs that aresubstantially free of abnormalities. The VAE model is trained viaunsupervised learning in which the model extracts and learns featuresand patterns of the normal training images. That is, the model mayanalyze raw normal image data to identify features and patterns ofnormal images without external identification. For example, usingbackpropagation, the unsupervised algorithm can continuously trainitself by setting the target output values to equal the inputs. Duringtraining, the VAE can compress (encode) an input normal training imageto a small vector of encodings from which it must contain enoughinformation to reconstruct the input image by the decoder. Subsequently,the decoder can expand (decode) the compressed form (output) toreconstruct the input image. The VAE may be trained by comparing thereconstructed output image and the input image in an iterative processto minimize the difference between them. By doing this, the autoencoderis forced to learn features about the image being compressed. Thus, theautoencoder learns features of normal images in an unsupervised manner.

When trained on only normal data, the resulting model is able to performefficient inference and to determine if a test image is normal or not,as well as to reconstruct an abnormal image into a corresponding normalimage. Therefore, once learned, the model can be given an image topredict whether it is normal or abnormal. Because the trained model hasbeen trained using only normal images, it can detect features (e.g.,abnormalities) different from the learned normal features (e.g.,representative of healthy subjects) as abnormal features.

Additionally, when given an abnormal image, such as a reference imageindicative of a particular severity level or an abnormal subject image,the trained model can generate a corresponding normal image. Forinstance, an abnormal image may be obtained and compressed with theencoder. Using the decoder, the input normal features (encodings) arerestored, and the input abnormal features are not restored. Because thetrained model has been trained on normal images, normal features of theinput abnormal image can be restored, but abnormal features cannot berestored. Rather, abnormal features in the input abnormality image aredeleted when the compressed image is restored, and the abnormalfeature(s) in the original (input) image are restored as normalfeature(s) in the restored image (generated normal image). As such, therestored image produced by the decoder from the compressed imagecorresponds to the input abnormality image except that an abnormality inthe input image is omitted.

Therefore, as discussed below, normal images may be generated fromreference images for each severity level by deep learning techniques.Similarly, a normal image may be generated from the subject image. Inother words, deep learning may be used to generate normal images thatcorrespond to reference images for each severity level or correspond tothe subject image except that abnormal part(s) have been removed. Then,the subject and reference difference images may be obtained by patternmatching. By using deep learning, a pseudo-normal image can beaccurately generated, and an accurate difference image can be generated.Additionally, it reduces the amount of data necessary for training thedeep learning device or network.

Difference images may be obtained by comparing a subject image and anormal image generated by deep learning. Reference difference images mayalso be obtained by comparing reference images indicative of variousseverity levels to normal images generated by deep learning techniques.The difference images may be raw difference images or they may beprocessed or manipulated to filter out noise or movement and increasethe dynamics of effect, e.g., of different pixel values to illustrateabnormalities or deviations in the area of interest. The differenceimages may be further processed to smooth out the image and remove highfrequency noise. For example, a lowpass spatial filter can block highspatial frequencies and/or low spatial frequencies to remove highfrequency noises at either end of the dynamic range. This provides asmoothed-out processed difference image (in digital format).

In another aspect, reference data, including image and non-image data,may be collected from people or groups of people. Such people mayinclude healthy people that are not suffering from a gastric condition,and other people suffering from various gastric conditions and severitylevels thereof, including, for example, gastric atrophy, gastritis, andgastric cancer. The reference image and non-image data may bestandardized and categorized according to one or more characteristics.For example, such reference data may be categorized based on populationcharacteristics, such as race, gender, or age of the people from whichthe data was collected. Standardized data permits average stomachcharacteristics to be calculated for healthy subjects and subjects withdifferent severity levels of each particular gastric condition.

An exemplary method 48 for generating abnormality images, indicative ofdifferences between a region of the subject's stomach and a referencestomach region, is illustrated in FIG. 3. Reference image data isacquired in step 50. Reference image data 50 may include standard imagedata of healthy, normal subjects, and reference image data of subjectswith various gastric conditions and severity levels thereof. Thereference image data may be categorized and standardized in step 52. Forexample, reference image data may be categorized and/or standardizedaccording to one or more desired characteristics, such as age, gender,or race. While the presently illustrated embodiment is described withrespect to image data, it is noted that reference non-image data andsubject non-image data may also, or instead, be used to generate theabnormality images discussed herein.

The method 48 may include a step 54 of selecting a subset of thereference image data based on a subject characteristic. For instance, ifa subject is a thirty-five year old Japanese man, a subset of thereference image data grouped to include reference images pertaining tomen between thirty and forty years of age may be more relevant forcomparative purposes than a group of reference images composed of datacollected from men between sixty and seventy years of age. Similarly, asubset of the reference image data grouped to include reference imagespertaining to Japanese men may be more relevant for comparative purposesthan a group of reference images composed of data collected fromCaucasian men. A subset of reference image data collected from Japanesemen between thirty and forty years of age may be the most relevant forcomparative purposes.

Once a desired group of reference image data is selected, the matchedreference image data 56 may be compared to image data 60 of the subjectin step 58. For example, the subject abnormality image may be adifference image between the reference image data 56 and the subjectimage data 60 may be created. The abnormality image may be generatedfrom a comparison of standard image data of normal, healthy subjects andthe subject image data. Non-image data of the subject may instead oralso be compared to matched reference non-image data, as describedabove. Additionally, the various data may be processed and categorizedin any suitable manner to facilitate such comparisons.

Additionally, reference data may be categorized and sorted intostandardized databases, such as through an exemplary method shown inFIG. 4. The method may include acquiring reference data 70, which mayinclude image and non-image data from various people, and categorizingthe data in step 72. For example, the reference data 70 may becategorized into various groups, such as normal (healthy) subject data74, data 76 of subjects clinically diagnosed with a first gastriccondition, data 78 of subjects diagnosed with a second gastriccondition, and data 80 of subjects diagnosed with a third condition.Such gastric conditions may include, for example, gastritis, includingatrophic gastritis, and gastric cancer. The reference data 70 mayfurther be categorized according to severity level for each gastriccondition, as discussed in more detail below. The data 74, 76, 78, and80 may be stored in respective databases 82, 84, 86, and 88. Suchdatabases may be stored on a server, in one or more memory or storagedevices, and/or in other suitable media. Such databases may becontinuously or periodically updated as more subjects are diagnosed witha particular gastric condition and severity level thereof.

Based on the subject and reference image and non-image data discussedabove, numerous reference and subject abnormality data and images may becreated. By way of example, an exemplary method 100 for generating andanalyzing such abnormality data is shown in FIG. 5. The method 100includes acquiring reference stomach data for: normal subjects withoutdiagnosed gastric conditions (data 104), subjects diagnosed withgastritis, including gastric atrophy (data 106), and subjects diagnosedwith gastric cancer (data 108). Each subset of reference stomach datafor a particular gastric condition may be further divided into groupsbased on severity level. The method 100 may also include acquiringsubject stomach data 110. The method 100 may acquire reference stomachdata for other gastric disorders, which may be processed in a mannersimilar to those discussed in the present example. Indeed, the presentprocessing techniques may also be applied to other disorders unrelatedto the stomach.

In step 112, the standard data 104 may be compared to each of the otherdata 106, 108, and 110, to generate gastritis abnormality data 118,gastric cancer abnormality data 120, and subject abnormality data 114,all of which may represent deviations from the standard/normal data 104.The abnormality data for each gastric condition may be further dividedinto groups based on severity level. Such abnormality data may includestructural abnormality images representative of differences between thesubject data and: (i) the reference data for the particular gastriccondition, and (ii) the normal reference data. For example, structuralabnormality images may include mucosa thickness images, blood vesselpermeation images, lesion size images, and lesion depth images.

In step 122, such abnormality data may be analyzed. For example, asubject abnormality image or data may be compared to representativereference abnormality images or data for each of the above noted gastricconditions to facilitate diagnosis of the subject with respect to one ormore of such gastric conditions and a severity level thereof.Additionally, reference clinical data 124, subject clinical data 126,and other data 128 may also be analyzed by a data processing system or auser to facilitate diagnosis. Such analysis may include pattern matchingof subject images and reference images, and confidence levels of suchmatching may be provided to a user. Finally, results 130 of the analysismay be output to storage or to a user via, for example, an output device18, such as a display or printer.

A method 130 for analyzing the data discussed above and diagnosing asubject is illustrated in FIG. 6. In step 132, one or more subjectabnormality images, which may include a structural abnormality image orsome other abnormality image, may be compared to one or more referenceabnormality images, such as those previously described. Notably, thereference abnormality images may include abnormality imagesrepresentative of one or more gastric conditions, as well as variousseverity levels of the one or more gastric conditions.

Based on such comparisons, one or more subject gastric conditions and/orseverity levels may be identified in step 134 and diagnosed in step 138.In some embodiments, such as a fully automated embodiment, steps 134 and136 may be combined. In other embodiments, however, the identificationand diagnosis may be performed as separate steps. For instance, the dataprocessing system 10 may identify various potential gastric conditionsand/or severity levels and present the identified conditions and/orseverity levels to a user for diagnosis. A report 138 may include anindication of the identified subject gastric condition(s) or severitylevels, the diagnosis, or both.

The extent of subject deviation from reference data may also betranslated into one or more abnormality scores, which may be generatedthrough the methods shown in FIGS. 7 and 8. An exemplary method 140 ofFIG. 7 may include accessing subject image data 142 and reference imagedata 144, including standard image data and image data representative ofa particular gastric condition and/or severity level thereof. Such imagedata may be received from any suitable source, such as a database or animaging system, such as an endoscope. The image data 142 and 144 mayinclude image data collected from a wide range of sources. The referenceimage data 144 may be standardized according to any desiredcharacteristics. For instance, the reference image data 144 may includedata representative of features of normal individuals with certaincharacteristics, for example, characteristics similar to the subject. Instep 146, the subject image data 142 and the reference image data 144may be compared to determine deviations of the subject image data 142from the reference image data 144. Such differences may generallyrepresent deviation, for example, structural differences between thesubject and normal (e.g., healthy) subjects.

The method 140 may also include calculating 148 one or more subjectimage abnormality scores for differences between the subject image data142 and the reference image data 144. Such abnormality scores may beindicative of an array of structural deviations of the subject relevantto the reference image data. The subject image abnormality scores may becalculated in various manners, such as based on projection deviation,single pixel (2D) deviation, single voxel (3D) deviation, or on anyother suitable technique. The calculated subject image abnormalityscores 150 may then be stored in a database 152, output to a user, ormay undergo additional processing in one or more further steps 154.

FIG. 8 shows an exemplary method 160 for calculating non-imageabnormality scores. The method 160 may include accessing subjectnon-image data 162 and reference non-image data 164. The non-image datamay be received from any suitable source, such as a database, acomputer, or subject monitor. The subject non-image data 162 may includeany non-image information collected for the purpose of diagnosing thesubject, such as clinical data, laboratory data, subject history, familyhistory, subject vital signs, and the like, and may also include resultsof other tests, such as genetic tests and so forth. The referencenon-image data 164 may include similar data, which may be standardizedbased on one or more characteristics of the persons from whom it wasobtained. The subject non-image data 162 and reference non-image data164 may include one or both of numeric data and enumerated data, each ofwhich may be continuous or discrete. The reference non-image data 164may include data representative of features of normal persons withparticular characteristics, such as those similar to the subject. Instep 166, the subject non-image data 162 may be compared to thereference non-image data 164 to identify differences between the data.Such differences may generally represent a deviation, such a structuraldeviation, of the subject from normal (e.g., healthy) individuals.

Additionally, the method 160 may include a step 168 of calculating oneor more subject non-image abnormality scores for differences between thesubject non-image data 162 and the reference non-image data 164. Varioustechniques may be used to calculate the subject non-image abnormalityscores, including, for example, z-score deviation or distributionanalysis. Of course, it will be appreciated that other calculationtechniques may also or instead be employed in other embodiments. Thecalculated subject non-image abnormality scores 170 may be stored in adatabase 172, output to a user, or may undergo additional processing inone or more further steps 174.

Subject abnormality scores may be used to generate one or more visualrepresentations to facilitate subject diagnosis. An exemplary method 180is illustrated in FIG. 9, which includes accessing one or more subjectimage abnormality scores and one or more subject non-image abnormalityscores in steps 182 and 184, respectively. These abnormality scores maybe processed 186 to generate a visual representation of the differencesrepresented by the subject abnormality scores. Subject abnormalityscores may be derived from dynamic data (e.g., video) or longitudinaldata (e.g., data acquired at discrete points in time over a givenperiod), and multiple visual representations corresponding to deviationsat different points of time may be generated in step 186. The one ormore visual representation may then be output 288 to facilitatediagnosis of the subject in step 190. For abnormalities derived fromdynamic or longitudinal data, multiple visual representations may beoutput simultaneously or sequentially.

Technical effects of the present disclosure include the accurate andconsistent diagnoses of various gastric conditions and severity levelsthereof, as well as providing decision support tools for user-diagnosisof subjects. For example, by using abnormality (e.g., difference)images, it may be easier to identify areas of abnormalities ordeviations from normal, as well as to determine the extent of deviationin a subject. For example, the difference images allow a user to focuson, extract, and enhance the deviations and abnormalities in the subjectimage that may not be as apparent from a comparison of the raw subjectimages with the standard images. Technical effects may also include thevisualization of subject image and non-image information together in aholistic, intuitive, and uniform manner, facilitating accurate andobjective diagnosis by a user. Additionally, the present systems,methods, and computer-readable media enable the generation of subjectabnormality images and reference abnormality images of known gastricconditions and/or severity levels thereof, and the combination of suchimages with non-image data, to facilitate quantitative assessment anddiagnosis of gastric conditions and their severity level. The disclosedsystems, methods, and computer-readable media enable analysis ofmultiple parameters, including both image and non-image data, toaccurately and objectively diagnose severity levels of gastricconditions.

A system may be programmed or otherwise configured to gather clinicalinformation and create integrated comprehensive views of the progressionof statistical deviations of data of an individual subject from one ormore normal subject populations over time from longitudinal data. Inother words, subject image and/or non-image data at a particular pointin time may be compared to subject image and/or non-image data collectedat an earlier point in time to determine a change in the data of thesubject over time. The change in the subject data over time may be usedto facilitate diagnosis, for example, diagnosis of gastric atrophy,gastritis, or gastric cancer and/or a severity thereof. In addition, thepresent systems, methods, and computer-readable media provide structuredintegrated comprehensive views of the deviation of the clinicalinformation across a given diseased subject population when comparedagainst a population of normal individuals, both at a single point intime and across multiple time points (longitudinally). Suchcomprehensive views described herein may display a normative comparisonto thousands of standardized and normalized data values concurrently.The resulting comprehensive view can provide patterns of deviations fromnormal that may indicate a characteristic pattern corresponding to knowngastric conditions or abnormalities and severity levels thereof.

Using the presently disclosed techniques, a user may be able to easilycompare the results of one parameter with another, and draw conclusionstherefrom. To facilitate such analysis, the various parameters may bestandardized and normalized. Further, an integrated comprehensive viewof clinical data of a specific population of people with respect to apopulation of normal subjects is provided. The view may includedisparate types of clinical data, including both image and non-imagedata in a manner that makes it easy for humans to distinguish thedistribution of clinical parameter results across gastric conditionpopulations. Although various graphs can be used to analyze results fora single clinical parameter across populations, they are quitecumbersome and impractical when it comes to visualizing and analyzing alarger number of parameters. The present disclosure analyzes multipleparameters, including both image and non-image data to accurately andobjectively diagnose severity levels of gastric conditions.

Stained Images to Facilitate Diagnosis of Severity of Gastric Atrophy

In an exemplary embodiment, a diagnostic system, method, andcomputer-readable storage medium for determining or facilitatingdiagnosis of a severity of gastric atrophy in a subject is provided.Prolonged inflammation causes normal stomach tissue to deform such thatthe surface, foveolar, and glandular epithelium in the oxyntic or antralmucosa is replaced by intestinal epithelium. This is a condition knownas intestinal metaplasia. Intestinal metaplasia results in the thinningof the stomach mucosa, which is known as atrophic gastritis or gastricatrophy. Progressive atrophy is believed to increase a subject's risk ofdeveloping gastric cancer. Staining the interior of the stomach makes itpossible to distinguish between a normal (e.g., healthy) stomach and astomach that has developed intestinal metaplasia.

In the present embodiment, the interior of the stomach of the subject isstained with a dye to obtain subject stained images, and the subjectstained images are compared with reference images to determine theseverity of gastric atrophy. Dye may be applied to the subject's stomachby any suitable method for staining a stomach. For example, the dye maybe sprayed onto the stomach using an endoscope or the subject may ingestthe dye before endoscopic images are taken. The dye may include anysuitable dye for staining the stomach, such as methylene blue, Evansblue, cardio blue, or brilliant blue. For example, methylene blue, Evansblue, or cardio blue may be sprayed onto the stomach by an endoscope tostain the subject's stomach blue. The subject may drink brilliant blueto stain the subject's stomach before collecting images. Then, anendoscope is inserted into the subject's stomach and stained images areacquired. The subject stained images may be stored on a server or in adatabase, for example, in the storage device 14, such as memory or otherstorage devices (FIG. 1).

Reference stained images (e.g., of stomachs of individuals withdifferent severity levels of gastric atrophy) and standard stainedimages (e.g., of healthy stomachs) may be obtained in the same mannerdiscussed above. For example, reference data, including stained imagedata and non-image data, may be collected from people or groups ofpeople. Such people may include healthy people that are not sufferingfrom gastric atrophy, and other people suffering from different severitylevels of gastric atrophy. The reference stained image and non-imagedata may be standardized and categorized according to one or morecharacteristics. For example, such reference data may be categorizedbased on population characteristics, such as race, gender, or age of thepeople from which the data was collected. Standardized data permitsaverage stomach characteristics to be calculated for healthy subjectsand subjects with different severity levels of gastric atrophy.

The standard stained image may be an earlier stained image of thesubject in a healthy state or may be a stained image of a stomach of adifferent person in a healthy state. As discussed in more detail below,the different person may be selected based on one or more sharedcharacteristics with the subject, such as age, race, and sex.

An abnormality stained image of the subject's stomach may be generatedthrough any suitable technique. For example, an abnormality image may begenerated by comparing the subject stained image with a standard stainedimage as discussed above. For example, a difference image between thesubject stained image and the standard stained image of a healthystomach may be obtained as a subject abnormality image. Prior tocomparing the subject stained image and the standard stained image, thesubject stained image and standard image may undergo preprocessing toextract or enhance certain areas or anatomical features, such as areasof thinning mucosa, according to any known methods. The images may alsobe standardized to facilitate comparison and analysis.

The abnormality image may be a representative image in which each pointof the image represents a score generally corresponding to a number ofstandard deviations (based on a selected population) in the differencebetween a subject value (e.g., staining intensity) and the average value(e.g., staining intensity) of the population for that point. Abnormalityimages may be generated from image data and/or one or more of numericaldata, text data, waveform data, image data, video data, and the like.

The image(s) may be visualized to facilitate further analysis ordiagnosis. For instance, any or all of the standard stained images,subject stained image, subject abnormality images, and reference stainedimages (discussed below) may be expressed as surface matrices, and canbe displayed or overlaid on a three-dimensional (3D) stomach surface.

An exemplary method 200 for generating abnormality images, indicative ofdifferences between a region of the subject stained stomach image and aregion of a standard stained stomach image, is illustrated in FIG. 10.Standard stained image data is obtained in step 202, and is categorizedand standardized in step 204. Standard stained image data and non-imagedata may be collected from people and categorized or standardizedaccording to one or more desired characteristics, such as age, gender,or race. While the presently illustrated embodiment is described withrespect to stained image data, it is noted that reference non-image dataand subject non-image data may also, or instead, be used to generate theabnormality images discussed herein.

The method 200 may include selecting a subset of the standard stainedimage data based on a subject characteristic, such as age, race, orgender in step 206, as discussed above with respect to FIG. 3 (e.g.,step 54). For example, if the subject is an eighty-five year oldJapanese woman, a subset of the standard image data grouped to includestandard stained images pertaining to women or Japanese women betweeneighty and ninety years of age may be selected comparative purposes asthis data may be more relevant than a group of standard stained imagescollected from other groups of individuals, such as men, or non-Japanesewomen, or younger individuals, such as individuals younger than eighty,seventy, or sixty.

Once a desired group of standard stained image data is selected, thematched standard stained image data 208 may be compared to stained imagedata 212 of the subject in step 210. Non-image data of the subject mayinstead or also be compared to matched standard non-image data, asdescribed above. Additionally, the various data may be processed andcategorized in any suitable manner to facilitate such comparisons. Instep 214, a subject abnormality image may be generated based at least inpart on the comparison 210 between the matched standard stained imagedata 208 and the subject stained image data 212.

Exemplary subject abnormality images are shown in FIGS. 11A-D. FIG. 11Ashows a healthy stomach with no signs of gastric atrophy. FIG. 11B showsan abnormality image of a stomach with low severity gastric atrophy.FIG. 11C shows an abnormality image of a stomach with moderate severitygastric atrophy. FIG. 11D shows an abnormality image of a stomach withhigh severity gastric atrophy. As shown in FIGS. 11A-11D, variousregions of the stomachs may be color coded according to a scale torepresent degree of atrophy, or deviation from normal mucosal thickness,to facilitate a user's understanding of the represented anatomicalinformation.

For example, the healthy stomach shown in FIG. 11A has almost nostaining, indicating that the mucosal thickness throughout the stomachis within healthy ranges and there are no signs of gastric atrophy. InFIG. 11B, the stomach abnormality image has minimal stained areas with alow staining intensity (e.g., lightly stained), indicating that themucosal thickness is slightly below healthy ranges in the lightlystained areas of the stomach, which corresponds to low severity gastricatrophy. In FIG. 11C, the stomach abnormality image has larger stainedareas than FIG. 11B, including areas with a higher staining intensity(e.g., medium staining intensity), indicating that the mucosal thicknessis slightly to moderately below healthy ranges in the stained areas(ranging from light to medium staining intensity), which corresponds tomoderate severity gastric atrophy. Lastly, FIG. 11D shows a stomachabnormality image that is almost entirely stained, including severalareas with a medium to high staining intensity, indicating that themucosal thickness is moderately to significantly below healthy ranges inthe stained areas (ranging from medium to high staining intensity),which corresponds to high severity gastric atrophy.

Additionally, reference stained image data may be categorized and sortedinto standardized databases, such as through an exemplary method 230shown in FIG. 12. The method may include acquiring reference data 232,which may include stained image and non-image data from various people,and categorizing the data in step 234. For example, the reference data232 may be categorized into various groups, such as normal (healthy)subject data 236, data 238 of subjects clinically diagnosed with lowseverity gastric atrophy, data 240 of subjects diagnosed with moderateseverity gastric atrophy, and data 242 of subjects diagnosed with highseverity gastric atrophy. The data 236, 238, 240, and 242 may be storedin respective databases 244, 246, 248, and 250. Such databases may bestored on a server, in one or more memory or storage devices, and/or inother suitable media. Such databases may be continuously or periodicallyupdated as more subjects are diagnosed. As discussed above, the data236, 238, 240, and 242 in each database 244, 246, 248, and 250 may befurther standardized and classified according to various subjectcharacteristics, such as age, gender, and race.

Exemplary methods for diagnosing a subject with a particular severity ofgastric atrophy based at least in part of the foregoing data is shown inFIGS. 13A-B. The method 260 may include obtaining stained image(s) ofthe stomach of the subject 262, and comparing the subject stainedimage(s) 262 with standard stained image(s) 264 of healthy stomachs togenerate a subject abnormality image 266, which is indicative ofdifferences between the subject stained image(s) and the standardstained image(s).

Further with respect to FIG. 13A, reference stained images 268 ofstomachs exhibiting different severity levels of gastric atrophy arealso compared 270 to the standard stained image(s) 264 to generatereference abnormality images, which are indicative of differencesbetween the reference stained images 268 and the standard stainedimage(s) 264. Reference stained images 268 may be standardized imagesacquired from a database of reference stained images 268 for eachseverity level of gastric atrophy collected from a particular group ofpeople diagnosed with a particular severity level of gastric atrophy, asdiscussed above. The reference stained images 268 may be the actualstained images, optionally processed to enhance or extract thestructural feature of interest, collected from the people of aparticular group or characteristic category. Alternatively, thereference stained images 268 may be average images created based on thedata collected from the people of a particular population diagnosed witha particular severity level of gastric atrophy. For example, arepresentative average stained stomach image for each severity level ofgastric atrophy may be generated. Thus, multiple representative oraverage stained images may be created for each severity level of gastricatrophy.

The subject abnormality image 266 is then compared 272 with thereference abnormality images 270. All of the images may be standardizedinto one or more common or similar formats to facilitate analysis andcomparison. The subject is diagnosed 276 as having a particular severitylevel of gastric atrophy based at least on part on the comparisonbetween the subject abnormality image the reference abnormality images.The diagnosis 276 may also be made by taking other data 274 and analysisinto consideration, including non-image data, such as clinical data,laboratory data, subject history, family history, subject vital signs,results of various tests (e.g., genetic tests), and any other relevantnon-image data. Based on the subject and reference data, numerousreference and subject abnormality data and images may be created. Then,a report 278 of the diagnosis 276 is output, for example, to an outputdevice 18 (shown in FIG. 1), such as a display or a printer, or to adatabase for storage, or to a user in a human-readable format.

The various images and data described herein may be stored in one ormore databases to facilitate subsequent data analysis. Moreover, any orall of the foregoing comparisons may be performed either automaticallyby a data processing system, such as system 10, or by a medicalprofessional, such as a doctor, or by some combination thereof, tofacilitate automatic or manual diagnosis of the subject in step 276.

FIG. 13B shows an exemplary process in which a subject is diagnosed withmoderate severity gastric atrophy based on a comparison of subjectstained images with reference images. One or more stained images of thesubject are compared with one or more normal stained images to generatea subject abnormality image. For instance, in FIG. 13B, the normalstained image is subtracted from the subject stained image to generate adifference image (subject abnormality image). As shown in FIG. 13B, thesubject abnormality image shows the staining differences between thesubject stained image and the normal stained image and eliminates sharedstaining between the images. The staining differences shown in thesubject abnormality image are indicative of abnormalities or deviationsfrom normal.

Reference abnormality images (representative of various severity levelsof gastric atrophy) are compared to the normal stained image in the samemanner to generate reference abnormality images (representative ofvarious severity levels of gastric atrophy). For example, in FIG. 13B,the normal image is subtracted from each reference image to generate acorresponding reference difference (abnormality) image. Then the subjectabnormality image is compared to the reference abnormality images tomake a diagnosis. Not only the area of staining (e.g., size), but alsothe staining intensities of the stained area in the abnormality imagesis compared. In FIG. 13B, the subject abnormality image most closelyresembles the reference abnormality image representative of moderateseverity gastric atrophy. Therefore, based at least in part on thiscomparison, the subject is diagnosed with moderate severity gastricatrophy in the example shown in FIG. 13B.

The generation of the subject and reference abnormality (e.g.,difference) images enables the abnormalities and deviations in thesubject stained image to be visualized, thereby facilitating accurateand consistent diagnoses to be made. Such differences cannot be aseasily detected by simply viewing and comparing the subject andreference images. Additionally, as discussed in more detail below, theimages can be analyzed for, for example, staining intensity to furtherensure objective and consistent diagnoses are made.

FIG. 13C shows an example where the subject difference image isgenerated from a subject stained image and a normal stained imagegenerated by deep learning. Similarly, as shown in FIG. 13D, referencedifference images may be generated from reference stained imagesrepresentative of various severity levels of gastric atrophy and normalimages generated by deep learning. That is, instead of comparing thesubject and reference stained images with the same normal (e.g.,standard) image, the present systems and methods employ deep learningtechniques to generate normal images for use in generating the referenceand subject differences images.

For example, as discussed above, a deep learning model, such as a VAE,may be trained to generate corresponding normal images. Such trainingmay be unsupervised learning in which only normal stained images areinput into the model for training. Such normal training images may bestained images of healthy stomachs that are substantially free ofabnormalities. The model may be trained via an iterative processinvolving compressing and reconstructing input normal stained images byan encoder and a decoder to extract and learn features and patterns ofnormal stained images without external identification.

Once learned, the model can be given a stained image of a stomach topredict whether it is normal or abnormal. Because the trained model hasbeen trained using only normal stained images, it can detect featuresdifferent from normal stained images as abnormal features. Additionally,when given an abnormal stained image, such as a reference imageindicative of a particular severity level or an abnormal subject image,the trained model can generate a corresponding normal stained image bycompressing the abnormal image and restoring the compressed image as anormal stained image in which the abnormal features are omitted. Thatis, the trained model generates a normal stained image from the abnormalimage. The abnormal feature(s) in the original image are restored asnormal feature(s) in the restored stained image (generated normalstained image).

Therefore, as illustrated in FIGS. 13C and 13D, a normal stained imagemay be generated from a subject stained image and normal stained imagesmay be generated from reference stained images for each severity levelby deep learning. In other words, deep learning may be used to generatenormal stained images that correspond to reference stained images foreach severity level or correspond to the subject stained image exceptthat abnormal part(s) have been removed. Then, the subject and referencedifference images may be obtained by pattern matching. By using deeplearning, a pseudo-normal stained image can be accurately generated, andan accurate difference image can be generated. Additionally, it issufficient to train the deep learning device or network using onlynormal training images, and it is not necessary to train with abnormalimages. Therefore, the amount of data necessary for training the deeplearning device or network can be reduced. Collecting a large amount ofabnormal data (e.g., abnormal images) is difficult. Thus, reducing theburden of collecting a large amount of data is a significant advantage.

Based on the subject and reference stained image and non-image datadiscussed above, numerous reference and subject abnormality data andimages may be created. By way of example, an exemplary method 280 forgenerating and analyzing such abnormality data is shown in FIG. 14. Themethod 280 includes acquiring reference stomach data, includingreference stained images, for: normal subjects without gastric atrophy(data 282) (e.g., standard data, including standard stained image(s)),subjects clinically diagnosed with low severity gastric atrophy (data284), subjects diagnosed with moderate severity gastric atrophy (data286), and subjects diagnosed with high severity gastric atrophy (data288). The method 280 may also include acquiring subject stomach data290, including subject stained images.

In step 292, the standard data 282 may be compared to each of the otherdata 284, 286, 288, and 290, to generate low severity gastric atrophyabnormality data 296, moderate severity gastric atrophy abnormality data298, high severity gastric atrophy abnormality data 300, and subjectabnormality data 294, all of which may represent deviations from thestandard/normal data 282. Such abnormality data may include structuralabnormality images representative of differences between the subjectdata 290 and: (i) the reference data 284, 286, and 288 for theparticular severity level of gastric atrophy, and/or (ii) the standarddata 282. Structural abnormality images may deviations in mucosalthickness from normal, healthy ranges for a particular population. Forexample, the thinner the mucosa, the more severe the atrophy. Asdiscussed above, increased areas of thinner mucosa may be indicative ofmore severe gastric atrophy, whereas a few or some areas of slightly tomoderately thinner mucosa may be indicative of low or moderate severitygastric atrophy.

In step 302, such abnormality data may be analyzed. For example, asubject abnormality image or data may be compared to representativereference abnormality images to facilitate diagnosis of the subject withrespect to a particular severity level of gastric atrophy. Additionally,reference clinical data 304, subject clinical data 306, and other data308 may also be analyzed by a data processing system or a user tofacilitate diagnosis. Such analysis may include pattern matching ofsubject images and reference images, and confidence levels of suchmatching may be provided to a user. Finally, results 310 of the analysismay be output to storage or to a user via, for example, an output device18, such as a display or printer.

The extent of subject deviation from standard data may also betranslated into one or more abnormality scores, which may be generatedthrough the methods shown in FIGS. 7 and 8, discussed above. Forexample, with reference to FIG. 7, the method 140 may be includeaccessing subject image data 142 (e.g., stained image data) andreference image data 144 (e.g., stained image data), including standardstained image data and stained image data representative of a particularseverity level of gastric atrophy. Such stained image data may bereceived from any suitable source, such as a database or an imagingsystem, such as an endoscope. The stained image data 142 and 144 mayinclude stained image data collected from a wide range of sources. Thereference stained image data 144 may be standardized according to anydesired characteristics. For instance, the reference stained image data144 may be standard stained image data generally representative offeatures of normal (e.g., healthy) individuals with certaincharacteristics, for example, characteristics similar to the subject. Instep 146, the subject stained image data 142 and the reference stainedimage data 144 may be compared to determine deviations of the subjectstained image data 142 from the reference stained image data 144. Suchdifferences may generally represent deviation, for example, structuraldifferences between the subject and normal (e.g., healthy) subjects.

The method 140 may also include calculating 148 one or more subjectstained image abnormality scores for differences between the subjectstained image data 142 and the reference stained image data 144. Suchabnormality scores may be indicative of an array of structuraldeviations of the subject relevant to the reference stained image data.The subject stained image abnormality scores may be calculated invarious manners according to any suitable technique. The calculatedsubject stained image abnormality scores 150 may then be stored in adatabase 152, output to a user, or may undergo additional processing inone or more further steps 154. The method 160 of FIG. 8 of calculating asubject non-image abnormality score, which is discussed in more detailabove, may also be calculated for diagnosing or facilitating diagnosisof a particular severity level of gastric atrophy.

The abnormality score may be generated through the exemplary method 400shown in FIG. 15. The method 400 of FIG. 15 includes dividing 402 thesubject stained abnormality image by a specified area. For example, thesubject stained abnormality image may be divided into 16×16 areas. Then,the staining intensity of each stained area of the image may be measuredin step 404 to determine a staining intensity score. The stainingintensity may be measured as, for example, a contrast area, such as ablue contrast area. Then, the measured staining intensity may becorrelated to a particular staining intensity score.

For example, the staining intensity for each area may be measured by animage analysis device. Likewise, the staining intensity score may bedetermined by the same image analysis device or any other device. Theimage analysis device or other device may be the processor 12 (FIG. 1)or another processor or device that is part of the system 10 shown inFIG. 1. For example, computer-readable instructions for analyzing imagesto measure staining intensity and/or assign a corresponding stainingintensity score for each image area may be stored in the storage device14, such as the memory or another storage device.

As an example, if the measured staining intensity of a particular areais 0-50, then the staining intensity score for that area may be 0.1points. If the measured staining intensity is 50-150, then the stainingintensity score for that area may be 0.5 points. If the measuredstaining intensity is 150-255, then the staining intensity score forthat area may be 0.8 points.

Then, in step 406, a position score for each stained area may bedetermined based on the position of each stained area. For example, ifthe stained area is positioned in a region known to be prone to atrophyor disease, then the stained area may be assigned a larger positionscore, whereas the position score may be lower if the stained area ispositioned in a region not known to be prone to atrophy or disease or ifthe stained area is known to be resistant to atrophy.

The position score for each area may be determined by an image analysisdevice or the processor 12 (FIG. 1) or another processor or device thatis part of the system 10 shown in FIG. 1. For example, the processor mayexecute computer-readable instructions and/or algorithms for analyzingimages to determine a position of each stained area and/or assign acorresponding position score for each stained area. Thecomputer-readably instructions may be stored in the storage device 14,such as the memory or another storage device.

As an example, the position factor may be 0.1 points for anatrophy-resistant area or an area not susceptible to atrophy, 0.5 pointsfor an area known to have a small to moderate likelihood of atrophy, and0.8 points for an area prone to atrophy.

In step 408, an abnormality score for each stained area may becalculated based on the staining intensity score determined in step 404and the position score determined in step 406. For example, the stainingintensity score 404 for each stained area may be multiplied by theposition score 406 for each stained area to calculate the abnormalityscore 408 for each stained area. As an example, for a certain area, ifthe measured staining intensity is 100, and the area is an area prone toatrophy, then the abnormality score may be calculated as 0.4. That is,based on the above exemplary staining intensity and positions scoreranges, the staining intensity score would be 0.5, and the positionfactor would be 0.8, resulting in an abnormality score of 0.5×0.8, whichis 0.4. In other words, the staining intensity score 404 of each stainedarea may be weighted based on its position 406.

The above staining intensity score ranges and position score ranges aremerely exemplary, non-limiting ranges. The staining intensity score andposition score ranges may be determined by the user or may be determinedbased on the reference and/or standard image and non-image data.

Then, in step 410, the overall abnormality score may be calculated as atotal of the abnormality scores 408 for the stained areas. In otherwords, the weighted score (abnormality score 408) for each stained areamay be combined to determine an overall stained image abnormality scorein step 410. For example, if the image was divided into 125 areas, andfifty areas were determined to have an abnormality score of 0.4, thirtyareas were determined to have an abnormality score of 0.05, twenty-fiveareas were determined to have an abnormality score of 0.08, ten areaswere determined to have an abnormality score of 0.64, and ten areas weredetermined to have an abnormality score of 0.01, the overall abnormalityscore may be calculated to be 30(=(0.4×50)+(0.05×30)+(0.08×25)+(0.64×10)+(0.01×10)).

Alternatively, the staining intensity score may be the abnormalityscore, eliminating steps 406 and 408 in FIG. 15. In other words, thestaining intensity score for each stained area determined in step 404may be the abnormality score of each stained area such that the overallabnormality score may be calculated in step 410 as a total of thestaining intensity scores for each area 404.

In any event, the overall abnormality score may be output 412 to theuser, for example, via an output device 18, such as a display or aprinter, or the overall abnormality score 410 may be output to adatabase or a server for storage. The overall abnormality score 410 mayalternatively undergo additional processing in step 414 before beingoutput to a user or server. Alternatively, a user may instruct theprocessor system 10 to perform additional processing 414 after output ofthe overall abnormality score 410.

Although the above description discloses calculating the abnormalityscore based on the subject abnormality image, the abnormality scorecould instead be calculated using the subject stained image or any otherimage derived from the subject stained image.

As discussed above with respect to FIG. 9, subject abnormality scoresmay be used to generate one or more visual representations to facilitatesubject diagnosis. For example, one or more subject image abnormalityscores and one or more subject non-image abnormality scores may beacquired and processed to generate a visual representation of thedifferences represented by the subject abnormality scores. Subjectabnormality scores may be derived from dynamic data (e.g., video) orlongitudinal data (e.g., data acquired at discrete points in time over agiven period), and multiple visual representations corresponding todeviations at different points of time may be generated. The one or morevisual representation may then be output to facilitate diagnosis of thesubject in step. For abnormalities derived from dynamic or longitudinaldata, multiple visual representations may be output simultaneously orsequentially.

Reference abnormality scores may be determined in the same manner forthe reference abnormality images indicative of different severity levelsof gastric atrophy. After dividing the reference abnormality images intospecified areas, calculating the abnormality score for each area basedon the staining intensity score and/or position score for each area, andcalculating the overall reference abnormality score for each referenceabnormality image, the reference abnormality scores may also be outputto a server or database for storage, or may be output to a user. Thereference abnormality scores may be calculated for several referenceimages representative of a particular severity level of gastric atrophyto determine a reference abnormality score range or average for eachseverity level of gastric atrophy. A reference abnormality score oraverage score or range for each particular severity level may bedetermined from a composite reference image generated from multiplereference abnormality images representative of the particular severitylevel of gastric atrophy.

As more subjects are diagnosed as having a particular severity level ofgastric atrophy using the system and methods disclosed herein, thereference image database and standard image database for subjects thatare determined to be healthy may be continuously or periodicallyupdated. Likewise, the reference abnormality score, average score, orscore range for each severity level of gastric atrophy may becontinuously or periodically updated based on updated reference data.

The present embodiment may further include systems and methods fordiagnosing a subject with a particular severity level of gastric atrophyand/or diagnosing a likelihood of stomach cancer based at least in partof the abnormality score. An exemplary method 500 is shown in FIG. 16.In particular, the method 500 includes a step 502 of selecting a subsetof reference data based on a subject characteristic, such as age,gender, or race, as discussed in more detail above. In step 504, arelationship is calculated between the abnormality scores and thegastric atrophy severity level of the matched reference data (e.g., thatshare a characteristic with the subject).

As an example, the matched reference data may include: (i) 10,000subjects with a score of 0-20 points, of which 100 have a large degreeof atrophy, and 50 have stomach cancer, (ii) 2,000 subjects with a scoreof 20-40 points, of which 600 have a large degree of atrophy, and 300have stomach cancer, and (iii) 500 subjects with a score of 40-60points, of which 400 have a large degree of atrophy, and 200 havestomach cancer. In this case, it may be determined that a subject with ascore of 0-20 points has low severity gastric atrophy and a 0.5%possibility of stomach cancer, a subject with a score of 20-40 pointshas moderate severity gastric atrophy and a 15% possibility of stomachcancer, and a subject with a score of 40-60 points has high severitygastric atrophy, and a 40% possibility of stomach cancer.

For example, Table 1 shows an example correlation between the score andthe likelihood of having a various severity level of gastric atrophy.For example, according to Table 1, if the subject has a score of 50, theprobability of being normal is 10% (=200/2000), the probability of lowdegree gastric atrophy is 20%, the probability of moderate degreegastric atrophy is 30%, and the probability of high degree gastricatrophy is 40%.

TABLE 1 Low Moderate High Score Number Normal Degree Degree Degree  0-205000 3000 (60%) 1500 (30%) 450 (9%)  50 (1%) 20-40 3000 2000 (67%)  500(17%)  400 (13%) 100 (3%) 40-60 2000  200 (10%)  400 (20%)  600 (30%) 800 (40%)

In step 506, the subject abnormality score is compared to the ReferenceData. Based at least in part on that comparison, the subject is thendiagnosed in step 508 as having a particular severity of gastric atrophyand/or the possibility of stomach cancer. For example, if the subjecthas an overall abnormality score of 30, then the subject may bediagnosed as having moderate severity gastric atrophy and a 15%possibility of stomach cancer. The diagnosis may be output to a user,such as a doctor or technician via a display or printer or other outputdevice 18, and/or the diagnosis may be stored on a server or databasein, for example, a storage device 14.

The exemplary diagnostic processor system 10 shown in FIG. 1 may be usedto facilitate diagnosis of the severity level of gastric atrophy of thesubject and/or possibility of stomach cancer. For example,computer-readable instructions for analyzing and/or processing stainedimages and data, generating abnormality images, calculating abnormalityscores, performing or facilitating diagnosis of a particular severitylevel of gastric atrophy and/or possibility of stomach cancer may bestored in the storage device 14, such as the memory. The processor 12may execute the computer-readable instructions to facilitate diagnosisof the subject. As an output device 18, a display may be configured todisplay one or more of: the subject stained image, the subjectabnormality image, the standard stained image, the reference stainedimages, the reference abnormality images, non-image subject data, asubject deviation score, and the diagnosis received from the processor.

Based on the diagnosis of the severity of gastric atrophy, the subjectmay be appropriately treated. The processor may determine whichtreatment is appropriate based on the severity level of gastric atrophyand output treatment information accordingly. For example, medicationsthat block acid production and promote healing may be administered. Suchmedications including proton pump inhibitors, such as omeprazole,lansoprazole, rabeprazole, esomeprazle, dexlansoprazole, andpantoprazole. Proton pump inhibitors reduce acid by blocking the actionof the parts of the cells that produce acid. Acid blockers or histamine(H-2) blockers may be administered to reduce the amount of acid releasedin the subject's digestive tract. Such acid blockers include ranitidine,famotidine, cimetidine, and nizatidine. Antacids may also beadministered to neutralize exciting stomach acid and provide painrelief. Additionally, antibiotic medications may be administered to killH. pylori in the subject's digestive tract. Such antibiotics may includeclarithromycin, amoxicillin, and metronidazole. Further, stomach coatingdrugs, such as bismuth subsalicylate, that help protect the tissues thatline the stomach and small intestine may be administered.

For example, in the treatment of acute gastritis (low degree gastricatrophy), elimination of the cause is important. If the cause is clear,such as stress or drug use, it may be treated by removing the cause. Ifnausea and vomiting are severe, fasting, feeding by drip infusion, andtreatment with gastric acid secretion inhibitors and gastric mucosalprotective agents may indicated. If there is bleeding in the gastricmucosa, then use of a hemostat may be indicated.

In the case of chronic gastritis (moderate severity gastric atrophy),the subject may be treated with a drug that suppresses gastric acidsecretion. It may be used in combination with gastric mucous membraneprotective drugs and stomach movement function improving drugs.

For atrophic gastritis (high severity gastric atrophy), removal of H.pylori is indicated. In the eradication therapy, two types ofantibacterial drugs and one type of proton pump inhibitor (a drug thatsuppresses the secretion of gastric acid) may be taken twice daily for 7days. This therapy may be used to eliminate at least about 70% ofbacteria. For remaining bacteria, the subject should be treated againwith a different combination of antibacterial agents (secondaryeradication therapy). This should eliminate about 90% of bacteria.

Blood Vessel Images to Facilitate Diagnosis of Severity of Gastritis

In an exemplary embodiment, a diagnostic system, method, andcomputer-readable storage medium for determining or facilitatingdiagnosis of a severity of gastritis in a subject is provided. Chronicinflammation of the stomach results in thinning of the stomach's mucosa,allowing submucosal blood vessels to be imaged, for example, by anendoscope. Progression of chronic gastritis leads to progression ofsuperficial gastritis, atrophic gastritis, and intestinal metaplasia,and can eventually lead to gastric cancer. The present systems, methods,and computer-readable media diagnose the severity of gastritis based onthe contrast between the stomach wall and a blood vessel in stomach wallimages.

In the case of a normal gastric mucosal structure shown in FIG. 17,since a mucosal layer of a stomach is increased in thickness, most oflight is absorbed or reflected in the mucosal layer. In contrast, in acase of a mucosal structure in which gastritis has progressed, athickness of the mucosal layer is decreased with a decrease in thenumber of gastric gland cells. The change of the internal structure ofthe gastric mucosa with the progression of gastritis results in changesin an endoscopic image. For example, lamina muscularis mucosae, which isnormally a color close to white, becomes transparent, and the color ofatrophic mucosa becomes a faded color compared to a normal part. In anarea where there is an atrophic mucosa, when a mucosal layer isdecreased in thickness with atrophy, a blood vessel of a submucosabecomes visible in an endoscopic image.

An image of the gastric wall of the subject is obtained using anendoscope. The image may be a color image that has a pixel level (pixelvalue) for each wavelength component of R (red), G (green), and B (blue)in each pixel position. Each value of RGB may be stored in memory. Theimage may be color converted such that only green (G) is extracted inorder to highlight blood vessels in the image. For example, an image maybe acquired and color converted into a green (G) wavelength componentimage using known conversion processes. The G component may be usedbecause it is close to an absorption wavelength band of hemoglobin inblood so that structural information of the intraluminal image, such asthe structure of a blood vessel in the mucous membrane, is properlyrepresented. The G-component image may then be processed to removenoise, enhance edges and lines, and sharpen the image. Then, asdiscussed below, the brightness value of green (G) may be calculated asthe luminance value.

Such images show a contrast between the gastric wall and a blood vesselof the subject. The subject contrast images are compared with referencecontrast images to determine the severity of gastritis based on a degreeof permeation of a blood vessel. The subject contrast images and/orreference contrast images may be stored on a server or in a database.

Reference contrast images (e.g., of stomachs of individuals withdifferent severity levels of gastritis) and standard contrast images(e.g., of healthy stomachs) may be obtained in the same manner discussedabove. These contrast images, like the subject contrast image(s), show acontrast between the gastric wall and a blood vessel of the subject. Forexample, reference data, including contrast image data and non-imagedata, may be collected from people or groups of people. Such people mayinclude healthy people that are not suffering from gastritis, and otherpeople suffering from different severity levels of gastritis. Thereference contrast image and non-image data may be standardized andcategorized according to one or more characteristics, as discussedabove. For example, such reference data may be categorized based onpopulation characteristics, such as race, gender, or age of the peoplefrom which the data was collected. Standardized data permits averagestomach characteristics to be calculated for healthy subjects andsubjects with different severity levels of gastritis.

The standard contrast image may be an earlier contrast image of thesubject in a healthy state or may be a contrast image of a stomach of adifferent person in a healthy state. As discussed in more detail below,the different person may be selected based on one or more sharedcharacteristics with the subject, such as age, race, and sex.

An abnormality contrast image of the subject's stomach may be generatedthrough any suitable technique. For example, an abnormality image may begenerated by comparing the subject contrast image with a standardcontrast image as discussed above. For example, a difference imagebetween the subject contrast image and the standard contrast image of ahealthy stomach may be obtained as a subject abnormality image. Prior tocomparing the subject contrast image and the standard contrast image,the subject contrast image and standard image may undergo preprocessingto extract or enhance certain areas or anatomical features, such asblood vessels, according to any known methods. The images may also bestandardized to facilitate comparison and analysis.

The abnormality image may be a representative image in which each pointof the image represents a score generally corresponding to a number ofstandard deviations (based on a selected population) in the differencebetween a subject value (e.g., contrast intensity) and the average value(e.g., contrast intensity) of the population for that point. Abnormalityimages may be generated from image data and/or one or more of numericaldata, text data, waveform data, image data, video data, and the like.

The image(s) may be visualized to facilitate further analysis ordiagnosis. For instance, any or all of the standard contrast images,subject contrast image, subject abnormality images, and referencecontrast images (discussed below) may be expressed as surface matrices,and can be displayed or overlaid on a three-dimensional (3D) stomachsurface.

An exemplary method 600 for generating abnormality images, indicative ofdifferences between a region of the subject contrast stomach image and aregion of a standard contrast stomach image, is illustrated in FIG. 17.Standard contrast image data (e.g., standard blood vessel image) isobtained in step 602, and is categorized and standardized in step 604.Standard contrast image data and non-image data may be collected frompeople and categorized or standardized according to one or more desiredcharacteristics, such as age, gender, or race. While the presentlyillustrated embodiment is described with respect to contrast image data,it is noted that reference non-image data and subject non-image data mayalso, or instead, be used to generate the abnormality images discussedherein.

The method 600 may include selecting a subset of the standard contrastimage data based on a subject characteristic, such as age, race, orgender in step 606, as discussed above with respect to FIG. 3 (e.g.,step 54) and 10 (e.g., step 206). Once a desired group of standardcontrast image data is selected, the matched standard contrast imagedata 608 may be compared to contrast image data 612 of the subject(e.g., subject blood vessel image data) in step 610. Non-image data ofthe subject may instead or also be compared to matched standardnon-image data, as described above. Additionally, the various data maybe processed and categorized in any suitable manner to facilitate suchcomparisons. In step 614, a subject abnormality image may be generatedbased at least in part on the comparison 610 between the matchedstandard contrast image data 608 and the subject contrast image data612.

Exemplary subject abnormality images are shown in FIGS. 18A-D. FIG. 18Ashows a healthy stomach with no signs of gastritis. FIG. 18B shows anabnormality image of a stomach with low severity gastritis (superficialgastritis). FIG. 18C shows an abnormality image of a stomach withmoderate severity gastritis (atrophic gastritis). FIG. 18D shows anabnormality image of a stomach with high severity gastritis (intestinalepithelialization/intestinal metaplasia). As shown in FIGS. 18A-18D, adegree of contrast of the blood vessels represents a degree ofpermeation of the blood vessel through the mucosa (e.g., visibility ofthe blood vessel through the mucosa), or deviation from normal, tofacilitate a user's understanding of the represented anatomicalinformation.

For example, in the healthy stomach shown in FIG. 18A, there is nocontrast between the blood vessels and the stomach wall, indicating thatthe mucosal thickness throughout the stomach is within healthy rangesand there are no signs of gastritis. In the stomach abnormality image ofFIG. 18B, there is some contrast between some blood vessels and thestomach wall, indicating that the mucosal thickness is slightly tomoderately below healthy ranges, which corresponds to low severitygastritis (e.g., superficial gastritis). In the stomach abnormalityimage of FIG. 18C, there is increased contrast between even more bloodvessels 634 and the stomach wall, indicating that the mucosal thicknessis moderately below healthy ranges, which corresponds to moderateseverity gastritis (atrophic gastritis). Lastly, FIG. 18D shows astomach abnormality image in which the blood vessels cannot be seen andthe stomach wall appears to be different from that in FIGS. 18A-D. Thismay be is indicative of intestinal epithelialization (intestinalmetaplasia) and is indicative of high severity gastritis.

Subject contrast images generated over a period of time may be comparedto determine a change in the stomach of the subject over time. Forexample, subject contrast images may be obtained, for example, once ayear, twice a year, every two years, or any other time period. When anew subject contrast image is collected, it may be compared with one ormore earlier subject contrast images to analyze changes in the subjectcontrast images over time. Difference (e.g., abnormality) images may begenerated by subtracting earlier subject contrast images from the latestsubject contrast image. Such difference images may serve to emphasizethe changes between the subject contrast images collected at differentpoints in time.

For example, the comparison may reveal that a contrast of the bloodvessels has increased compared to the earlier contrast image, indicatingthat gastritis has advanced or become more severe. Alternatively, if theearlier subject image showed a high contrast between blood vessels andthe stomach wall, and a later subject image no longer shows a highcontrast or no longer shows blood vessel at all, it may be determinedthat intestinal epithelialization (intestinal metaplasia) has occurred(e.g., FIG. 18D).

Additionally, reference contrast image data may be categorized andsorted into standardized databases, such as through an exemplary method700 shown in FIG. 19. The method 700 may include acquiring referencedata 702, which may include contrast image and non-image data fromvarious people, and categorizing the data in step 704. For example, thereference data 702 may be categorized into various groups, such asnormal (healthy) subject data 706, data 708 of subjects clinicallydiagnosed with low severity gastritis, data 710 of subjects diagnosedwith moderate severity gastritis, and data 712 of subjects diagnosedwith high severity gastritis. The data 706, 708, 710, and 712 may bestored in respective databases 714, 716, 718, and 720. Such databasesmay be stored on a server, in one or more memory or storage devices,and/or in other suitable media. Such databases may be continuously orperiodically updated as more subjects are diagnosed. As discussed above,the data 706, 708, 710, and 712 in each database 714, 716, 718, and 720may be further standardized and classified according to various subjectcharacteristics, such as age, gender, and race.

Exemplary methods for diagnosing a subject with a particular severity ofgastritis based at least in part of the foregoing data is shown in FIGS.20A-B. For example, in FIG. 20A, the method 800 may include obtainingcontrast image(s) of the stomach of the subject 802, and comparing thesubject contrast image(s) 802 with standard contrast image(s) 804 ofhealthy stomachs to generate a subject abnormality image 806, which isindicative of differences between the subject contrast image(s) and thestandard contrast image(s).

Further with respect to FIG. 20A, reference contrast images 808 ofstomachs exhibiting different severity levels of gastritis are alsocompared 810 to the standard contrast image(s) 804 to generate referenceabnormality images, which are indicative of differences between thereference contrast images 808 and the standard contrast image(s) 804.Reference contrast images 808 may be standardized images acquired from adatabase of reference contrast images 808 for each severity level ofgastritis collected from a particular group of people diagnosed with aparticular severity level of gastritis, as discussed above. Thereference contrast images 808 may be the actual contrast images,optionally processed to enhance or extract the structural feature ofinterest, collected from the people of a particular group orcharacteristic category. Alternatively, the reference contrast images808 may be average images created based on the data collected from thepeople of a particular population diagnosed with a particular severitylevel of gastric atrophy. For example, a representative average contraststomach image for each severity level of gastritis may be generated.Thus, multiple representative or average contrast images may be createdfor each severity level of gastritis.

The subject abnormality image 806 is then compared 812 with thereference abnormality images 810. All of the images may be standardizedinto one or more common or similar formats to facilitate analysis andcomparison. The subject is diagnosed 814 as having a particular severitylevel of gastritis based at least on part on the comparison between thesubject abnormality image the reference abnormality images. Thediagnosis 814 may also be made by taking other data 816 and analysisinto consideration, including non-image data, such as clinical data,laboratory data, subject history, family history, subject vital signs,results of various tests (e.g., genetic tests), and any other relevantnon-image data. Based on the subject and reference data, numerousreference and subject abnormality data and images may be created. Then,a report 818 of the diagnosis 814 is output, for example, to an outputdevice 18 (shown in FIG. 1), such as a display or a printer, or to adatabase or server for storage, or to a user in a human-readable format.

FIG. 20B shows an exemplary process in which a subject is diagnosed withmoderate severity gastritis based on a comparison of subject contrastimages with reference contrast images. One or more blood vessel contrastimages of the subject are compared with one or more standard contrastimages to generate a subject abnormality image. For instance, in FIG.20B, the standard contrast image is subtracted from the subject contrastimage to generate a difference image (subject abnormality image). Asshown in FIG. 20B, the subject abnormality image shows the contrastdifferences between the subject contrast image and the standard contrastimage and eliminates shared contrast between the images. The contrastdifferences shown in the subject abnormality image are indicative ofabnormalities or deviations from normal.

Reference abnormality images (representative of various severity levelsof gastritis) are compared to the standard contrast image in the samemanner to generate reference abnormality images (representative ofvarious severity levels of gastritis). For example, in FIG. 20B, thestandard contrast image is subtracted from each reference contrast imageto generate a corresponding reference difference (abnormality) image.Then the subject abnormality image is compared to the referenceabnormality images to make a diagnosis. Not only the area or amount ofluminance, but also the intensities (saturation) of the contrast areasin the abnormality images are compared. In FIG. 20B, the subjectabnormality image most closely resembles the reference abnormality imagerepresentative of moderate severity gastritis. Therefore, based at leastin part on this comparison, the subject is diagnosed with moderateseverity gastritis in the example shown in FIG. 20B.

The generation of the subject and reference abnormality (e.g.,difference) images enables the abnormalities and deviations in thesubject contrast image to be visualized, thereby facilitating accurateand consistent diagnoses to be made. Such differences cannot be aseasily detected by simply viewing and comparing the subject andreference images. Additionally, as discussed in more detail below, theimages can be analyzed for, for example, contrast intensity to furtherensure objective and consistent diagnoses are made.

FIG. 20C shows an example where the subject difference image isgenerated from a subject blood vessel image and a normal blood vesselimage generated by deep learning techniques. Similarly, as shown in FIG.20D, reference difference images may be generated from reference bloodvessel images representative of various severity levels of gastritis andnormal blood vessel images selected by deep learning techniques. Thatis, instead of comparing the subject and reference images with the samenormal (e.g., standard) image, the present systems and methods employdeep learning techniques to generate normal blood vessel images for usein generating the reference and subject differences images.

For example, as discussed above, a deep learning model, such as a VAE,may be trained to generate corresponding normal images. Such trainingmay be unsupervised learning in which only normal blood vessel imagesare input into the model for training. Such normal training images maybe blood vessel images of healthy stomachs that are substantially freeof abnormalities. The model may be trained via an iterative processinvolving compressing and reconstructing input normal blood vesselimages by an encoder and a decoder to extract and learn features andpatterns of normal blood vessel images without external identification.

Once learned, the model can be given a blood vessel image of a stomachto predict whether it is normal or abnormal. Because the trained modelhas been trained using only normal blood vessel images, it can detectfeatures different from normal blood vessel images as abnormal features.Additionally, when given an abnormal blood vessel image, such as areference image indicative of a particular severity level or an abnormalsubject image, the trained model can generate a corresponding normalblood vessel image by compressing the abnormal image and restoring thecompressed image as a normal blood vessel image in which the abnormalfeatures are omitted. That is, the trained model generates a normalblood vessel image from the abnormal image. The abnormal feature(s) inthe original image are restored as normal feature(s) in the restoredblood vessel image (generated normal blood vessel image).

Therefore, as illustrated in FIGS. 20C and 20D, a normal blood vesselimage may be generated from a subject blood vessel image and normalblood vessel images may be generated from reference blood vessel imagesfor each severity level by deep learning. In other words, deep learningmay be used to generate normal blood vessel images that correspond toreference blood vessel images for each severity level or correspond tothe subject blood vessel image except that abnormal part(s) have beenremoved. Then, the subject and reference difference images may beobtained by pattern matching. By using deep learning, a pseudo-normalblood vessel image can be accurately generated, and an accuratedifference image can be generated. Additionally, it is sufficient totrain the deep learning device or network using only normal trainingimages, and it is not necessary to train with abnormal images.Therefore, the amount of data necessary for training the deep learningdevice or network can be reduced. Collecting a large amount of abnormaldata (e.g., abnormal images) is difficult. Thus, reducing the burden ofcollecting a large amount of data is a significant advantage.

The various images and data described herein may be stored in one ormore databases to facilitate subsequent data analysis. Moreover, any orall of the foregoing comparisons may be performed either automaticallyby a data processing system, such as system 10, or by a medicalprofessional, such as a doctor, or by some combination thereof, tofacilitate automatic or manual diagnosis of the subject in step 814.

Based on the subject and reference contrast image and non-image datadiscussed above, numerous reference and subject abnormality data andimages may be created. By way of example, an exemplary method 820 forgenerating and analyzing such abnormality data is shown in FIG. 21. Themethod 820 includes acquiring reference stomach data, includingreference contrast images, for: normal subjects without gastritis (data822) (e.g., standard data, including standard contrast image(s)),subjects clinically diagnosed with low severity gastritis (data 824),subjects diagnosed with moderate severity gastritis (data 826), andsubjects diagnosed with high severity gastritis (data 828). The method820 may also include acquiring subject stomach data 830, includingsubject contrast images.

In step 832, the standard data 822 may be compared to each of the otherdata 824, 826, 828, and 830, to generate low severity gastritisabnormality data 836, moderate severity gastritis abnormality data 838,high severity gastritis abnormality data 840, and subject abnormalitydata 834, all of which may represent deviations from the standard/normaldata 822. Such abnormality data may include structural abnormalityimages representative of differences between the subject data 830 and:(i) the reference data 824, 826, and 828 for the particular severitylevel of gastritis, and/or (ii) the standard data 822.

Structural abnormality images may show deviations in blood vesselvisibility (e.g., permeation) through mucosa (e.g., blood vesselcontrast), or mucosal thickness from normal, healthy ranges for aparticular population. For example, the higher the blood vessel contrast(e.g., the higher the blood vessel visibility/permeation through themucosa), or the thinner the mucosa, the more severe the atrophy,indicative of low or moderate gastritis. On the other hand, no bloodvessel contrast is indicative of a normal, healthy stomach. Thestructural abnormality images may also show deviations in surface,foveolar, and glandular epithelium in the oxyntic or antral mucosa, suchas replacement by intestinal epithelium, which is indicative of severegastritis.

In step 842, such abnormality data may be analyzed. For example, asubject abnormality image or data may be compared to representativereference abnormality images to facilitate diagnosis of the subject withrespect to a particular severity level of gastritis. Additionally,reference clinical data 844, subject clinical data 846, and other data848 may also be analyzed by a data processing system or a user tofacilitate diagnosis. Such analysis may include pattern matching ofsubject images and reference images, and confidence levels of suchmatching may be provided to a user. Finally, results 850 of the analysismay be output to a database or server, a storage device, and/or to auser via, for example, an output device 18, such as a display orprinter.

The extent of subject deviation from standard data may also betranslated into one or more abnormality scores, which may be generatedthrough the methods shown in FIGS. 7 and 8, discussed above. Forexample, with reference to FIG. 7, the method 140 may be includeaccessing subject image data 142 (e.g., contrast image data) andreference image data 144 (e.g., contrast image data), including standardcontrast image data and contrast image data representative of aparticular severity level of gastritis. Such contrast image data may bereceived from any suitable source, such as a database or an imagingsystem, such as an endoscope. The contrast image data 142 and 144 mayinclude contrast image data collected from a wide range of sources. Thereference contrast image data 144 may be standardized according to anydesired characteristics. For instance, the reference contrast image data144 may be standard contrast image data generally representative offeatures of normal (e.g., healthy) individuals with certaincharacteristics, for example, characteristics similar to the subject. Instep 146, the subject contrast image data 142 and the reference contrastimage data 144 may be compared to determine deviations of the subjectcontrast image data 142 from the reference contrast image data 144. Suchdifferences may generally represent deviation, for example, structuraldifferences between the subject and normal (e.g., healthy) subjects.

The method 140 may also include calculating 148 one or more subjectcontrast image abnormality scores for differences between the subjectcontrast image data 142 and the reference contrast image data 144. Suchabnormality scores may be indicative of an array of structuraldeviations of the subject relevant to the reference contrast image data.The subject contrast image abnormality scores may be calculated invarious manners according to any suitable technique. The calculatedsubject contrast image abnormality scores 150 may then be stored in adatabase 152, output to a user, or may undergo additional processing inone or more further steps 154. The method 160 of FIG. 8 of calculating asubject non-image abnormality score, which is discussed in more detailabove, may also be calculated for diagnosing or facilitating diagnosisof a particular severity level of gastritis.

The abnormality score may be generated through the exemplary method 900shown in FIG. 22. The method 900 of FIG. 22 includes dividing 902 thesubject blood vessel abnormality image by a specified area. For example,the subject abnormality image may be divided into 16×16 areas. Then, theluminance value of each contrast area of the image may be measured instep 904 to determine a luminance score.

The luminance value may be measured by color converting the image toextract only green (G) and measuring the brightness value of green (G)as the luminance value. Then, the measured luminance value may becorrelated to a particular luminance score.

For example, the luminance value for each contrast area may be measuredby an image analysis device. Likewise, the luminance score may bedetermined by the same image analysis device or any other device. Theimage analysis device or other device may be the processor 12 (FIG. 1)or another processor or device that is part of the system 10 shown inFIG. 1. For example, computer-readable instructions for analyzing imagesto measure luminance values and/or assign a corresponding luminancescore for each image area may be stored in the storage device 14, suchas the memory or another storage device.

As an example, if the measured luminance value of a particular area is0-50, then the luminance score for that area may be 0.1 points. If themeasured luminance value is 50-150, then the luminance score for thatarea may be 0.5 points. If the measured luminance value is 150-255, thenthe luminance score for that area may be 0.8 points.

Then, in step 906, a position score for each contrast area may bedetermined based on the position of each contrast area. For example, ifthe contrast area is positioned in a region known to be prone togastritis, then the contrast area may be assigned a larger positionscore, whereas the position score may be lower if the contrast area ispositioned in a region not known to be prone to gastritis or if thecontrast area is in a region is known to be resistant to gastritis.

The position score for each area may be determined by an image analysisdevice or the processor 12 (FIG. 1) or another processor or device thatis part of the system 10 shown in FIG. 1. For example, the processor mayexecute computer-readable instructions and/or algorithms for analyzingimages to determine a position of each contrast area and/or assign acorresponding position score for each contrast area. Thecomputer-readably instructions may be stored in the storage device 14,such as the memory or another storage device.

As an example, the position factor may be 0.1 points for an area notsusceptible to gastritis, 0.5 points for an area known to have a smallto moderate susceptibility of gastritis, and 0.8 points for an areasusceptible to gastritis.

In step 908, an abnormality score for each area may be calculated basedon the luminance score determined in step 904 and/or the position scoredetermined in step 906. For example, the luminance score 904 for eacharea may be multiplied by the position score 906 for each area tocalculate the abnormality score 908 for each area. As an example, for acertain area, if the measured luminance is 200, and the area is in aregion susceptible to gastritis, then the abnormality score may becalculated as 0.64. That is, based on the above exemplary luminancevalue and position score ranges, the luminance score would be 0.8, andthe position score would be 0.8, resulting in an abnormality score of0.5×0.8, which is 0.64. In other words, the luminance score 904 of eacharea may be weighted based on its position 906.

The above luminance score ranges and position score ranges are merelyexemplary, non-limiting ranges. The luminance score and position scoreranges may be determined by the user or may be determined based on thereference and/or standard image and non-image data.

Then, in step 910, the overall abnormality score may be calculated as atotal of the abnormality scores 908 for the areas. In other words, theweighted score (abnormality score 908) for each area may be combined todetermine an overall contrast image abnormality score in step 910. Forexample, if the image was divided into 145 areas, and ten areas weredetermined to have an abnormality score of 0.4, forty areas weredetermined to have an abnormality score of 0.05, fifteen areas weredetermined to have an abnormality score of 0.08, zero areas weredetermined to have an abnormality score of 0.64, and eighty areas weredetermined to have an abnormality score of 0.01, the overall abnormalityscore may be calculated to be 8(=(0.4×10)+(0.05×40)+(0.08×15)+(0.64×0)+(0.01×80)).

Alternatively, the luminance score may be the abnormality score,eliminating steps 906 and 908 in FIG. 22. In other words, the luminancescore for each area determined in step 904 may be the abnormality scoreof each area such that the overall abnormality score may be calculatedin step 910 as a total of the luminance scores for each area 904.

In any event, the overall abnormality score may be output 912 to theuser, for example, via an output device 18, such as a display or aprinter, or the overall abnormality score 910 may be output to adatabase or a server for storage. The overall abnormality score 910 mayalternatively undergo additional processing in step 914 before beingoutput to a user or server. Alternatively, a user may instruct theprocessor system 10 to perform additional processing 914 after output ofthe overall abnormality score 910.

Although the above description discloses calculating the abnormalityscore based on the subject abnormality image, the abnormality scorecould instead be calculated using the subject contrast image or anyother image derived from the subject contrast image.

As discussed above with respect to FIG. 9, subject abnormality scoresmay be used to generate one or more visual representations to facilitatesubject diagnosis. For example, one or more subject image abnormalityscores and one or more subject non-image abnormality scores may beacquired and processed to generate a visual representation of thedifferences represented by the subject abnormality scores. Subjectabnormality scores may be derived from dynamic data (e.g., video) orlongitudinal data (e.g., data acquired at discrete points in time over agiven period), and multiple visual representations corresponding todeviations at different points of time may be generated. The one or morevisual representation may then be output to facilitate diagnosis of thesubject in step. For abnormalities derived from dynamic or longitudinaldata, multiple visual representations may be output simultaneously orsequentially.

Reference abnormality scores may be determined in the same manner forthe reference abnormality images indicative of different severity levelsof gastritis. After dividing the reference abnormality images or otherreference images into specified areas, calculating the abnormality scorefor each area based on the luminance score and/or position score foreach area, and calculating the overall reference abnormality score foreach reference abnormality image, the reference abnormality scores mayalso be output to a server or database for storage, or may be output toa user. The reference abnormality scores may be calculated for severalreference images representative of a particular severity level ofgastritis to determine a reference abnormality score range or averagefor each severity level of gastritis. In other embodiments, a referenceabnormality score or average score or range for each particular severitylevel may be determined from a composite reference image generated frommultiple reference abnormality images representative of the particularseverity level of gastritis.

As more subjects are diagnosed as having a particular severity level ofgastritis using the system and methods disclosed herein, the referenceimage database and standard image database for subjects that aredetermined to be healthy may be continuously or periodically updated.Likewise, the reference abnormality score, average score, or score rangefor each severity level of gastritis may be continuously or periodicallyupdated based on updated reference data.

The present embodiment may further include systems and methods fordiagnosing a subject with a particular severity level of gastritisand/or diagnosing a likelihood of stomach cancer based at least in partof the abnormality score. An exemplary method 950 is shown in FIG. 23.In particular, the method 950 includes a step 952 of selecting a subsetof reference data based on a subject characteristic, such as age,gender, or race, as discussed in more detail above. In step 954, arelationship is calculated between the abnormality scores and thegastritis severity level of the matched reference data (e.g., that sharea characteristic with the subject).

As an example, the matched reference data may include: (i) 10,000subjects with a score of 0-20 points, of which 100 have a large degreeof gastritis, and 50 have stomach cancer, (ii) 2,000 subjects with ascore of 20-40 points, of which 600 have a large degree of gastritis,and 300 have stomach cancer, and (iii) 500 subjects with a score of40-60 points, of which 400 have a large degree of gastritis, and 200have stomach cancer. In this case, it may be determined that a subjectwith a score of 0-20 points has low severity gastritis and a 0.5%possibility of stomach cancer, a subject with a score of 20-40 pointshas moderate severity gastritis and a 15% possibility of stomach cancer,and a subject with a score of 40-60 points has high severity gastritis,and a 40% possibility of stomach cancer.

For example, Table 2 shows an example correlation between the score andthe likelihood of having a various severity levels of gastritis. Forexample, according to Table 2, if the subject has a score of 50, theprobability of being normal is 10% (=200/2000), the probability of lowdegree gastritis is 20%, the probability of moderate degree gastritis is30%, and the probability of high degree gastritis is 40%.

TABLE 2 Low Moderate High Score Number Normal Degree Degree Degree  0-205000 3000 (60%) 1500 (30%) 450 (9%)  50 (1%) 20-40 3000 2000 (67%)  500(17%)  400 (13%) 100 (3%) 40-60 2000  200 (10%)  400 (20%)  600 (30%) 800 (40%)

In step 956, the subject abnormality score is compared to the Referencedata. Based at least in part on that comparison, the subject is thendiagnosed in step 958 as having a particular severity of gastritisand/or the possibility of stomach cancer. For example, if the subjecthas an overall abnormality score of 8, then the subject may be diagnosedas having low severity gastritis and a 0.5% possibility of stomachcancer. The diagnosis may be output to a user, such as a doctor ortechnician via a display or printer or other output device 18, and/orthe diagnosis may be output to a server or database in, for example, astorage device 14.

The exemplary diagnostic processor system 10 shown in FIG. 1 may be usedto facilitate diagnosis of the severity level of gastritis of thesubject and/or possibility of stomach cancer. For example,computer-readable instructions for analyzing and/or processing contrastimages and data, generating abnormality images, calculating abnormalityscores, performing or facilitating diagnosis of a particular severitylevel of gastritis and/or possibility of stomach cancer may be stored inthe storage device 14, such as the memory. The processor 12 may executethe computer-readable instructions to facilitate diagnosis of thesubject. As an output device 18, a display may be configured to displayone or more of: the subject contrast image, the subject abnormalityimage, the standard contrast image, the reference contrast images, thereference abnormality images, non-image subject data, a subjectabnormality score, and the diagnosis received from the processor.

Based on the diagnosis of the severity of gastritis, the subject may beappropriately treated. The processor may determine which treatment isappropriate based on the severity level of gastritis and outputtreatment information accordingly. For example, medications that blockacid production and promote healing may be administered. Suchmedications including proton pump inhibitors, such as omeprazole,lansoprazole, rabeprazole, esomeprazle, dexlansoprazole, andpantoprazole. Proton pump inhibitors reduce acid by blocking the actionof the parts of the cells that produce acid. Acid blockers or histamine(H-2) blockers may be administered to reduce the amount of acid releasedin the subject's digestive tract. Such acid blockers include ranitidine,famotidine, cimetidine, and nizatidine. Antacids may also beadministered to neutralize exciting stomach acid and provide painrelief. Additionally, antibiotic medications may be administered to killH. pylori in the subject's digestive tract. Such antibiotics may includeclarithromycin, amoxicillin, and metronidazole. Further, stomach coatingdrugs, such as bismuth subsalicylate, that help protect the tissues thatline the stomach and small intestine may be administered.

For example, in the treatment of acute gastritis (low degree gastritis),elimination of the cause is important. If the cause is clear, such asstress or drug use, it may be treated by removing the cause. If nauseaand vomiting are severe, fasting, feeding by drip infusion, andtreatment with gastric acid secretion inhibitors and gastric mucosalprotective agents may indicated. If there is bleeding in the gastricmucosa, then use of a hemostat may be indicated.

In the case of chronic gastritis (moderate severity gastritis), thesubject may be treated with a drug that suppresses gastric acidsecretion. It may be used in combination with gastric mucous membraneprotective drugs and stomach movement function improving drugs.

For atrophic gastritis (high severity gastritis), removal of H. pyloriis indicated. In the eradication therapy, two types of antibacterialdrugs and one type of proton pump inhibitor (a drug that suppresses thesecretion of gastric acid) may be taken twice daily for 7 days. Thistherapy may be used to eliminate at least about 70% of bacteria. Forremaining bacteria, the subject should be treated again with a differentcombination of antibacterial agents (secondary eradication therapy).This should eliminate about 90% of bacteria.

Diagnosing the Severity of Gastric Cancer

In an exemplary embodiment, the diagnostic system, method, andcomputer-readable storage medium are designed for determining orfacilitating diagnosis of the severity of gastric cancer in a subject.

Gastric cancer is a disease in which malignant (cancer) cells form inthe stomach wall. As shown in FIG. 24, the wall 970 of the stomach ismade up of four layers of tissue. From the innermost layer to theoutermost layer, the layers of the stomach wall are: mucosa 972,submucosa 974 (including upper 972 a and lower 972 b mucosa), muscle976, and serosa 980, which includes subserosa (connective tissue) 978.Gastric cancer typically begins in the mucosa 972 and spreads throughthe outer layers as it grows. For instance, FIG. 24 shows various tumors988, 990, 992, 994, 996 penetrating different layers of the stomachwall. However, some gastric cancers, such as Scirrhous stomach cancer,may have lesion(s) 998 that are present in the submucosal layer 974 ormuscle layer 976, but are not present in the mucosal layer 972.

The severity of gastric cancer is generally expressed in stages ofgastric cancer. The stage of gastric cancer is representative of theextent of the cancer in the body, and is usually determined based on thedepth of gastric cancer in the stomach wall and the presence or absenceof metastasis. Determining the severity or stage of gastric cancer isuseful for determining how to treat the patient. Gastric cancer stagesrange from 0 to IV, with stage 0 being the earliest stage (least severe)stage IV being the most severe. In general, the lower the number, theless the cancer has spread. The staging system typically used forgastric cancer is the American Joint Committee on Cancer (ADCC) TNMsystem. The stage of gastric cancer is usually determined based on 3important pieces of information: (1) the tumor invasion depth of thecancer into the layers of the stomach wall (see e.g., FIG. 24), (2)whether the cancer has spread to nearby lymph nodes, and (3) whether thecancer as metastasized to distant sites (e.g., distant lymph nodes ororgans, such as the liver or lungs).

Gastric cancer is commonly detected and diagnosed by endoscopy in whichan image of the stomach is created by capturing the reflected light whenthe tissue is illuminated by light. Physicians typically diagnosegastric cancer and/or a severity (e.g., stage) of gastric cancer basedon structural changes (e.g., surface irregularities) of the surface ofthe stomach wall and changes in blood vessels by performing endoscopy toimage the stomach wall. A physician may first confirm the unevenness andcolor of the surface by obtaining a white light endoscopic image of thepatient's stomach wall. Then, the physician may confirm the blood vesselstructure in the stomach wall by obtaining an endoscopic image using anarrower band of illumination light, e.g., by Narrow Band Imaging (NBI).

Gastric cancer diagnoses and stage determinations, however, areinherently subjective determinations by physicians. For example,diagnoses may vary depending on the experience and knowledge of thephysician, resulting in inconsistent diagnoses. Additionally, diagnosesmay be made based on incomplete information. For example, in white lightimages, the unevenness of the tissue surface can be visualized, but theorganization (e.g., blood vessels) of the stomach wall cannot bevisualized. On the other hand, images acquired with narrower wavelengthbands can show tissue (e.g., blood vessels) within the layers of thestomach wall depending on the wavelength band of illumination light.However, the unevenness of the tissue surface cannot be visualized inNBI images. Thus, diagnoses based on only white light images or only NBIimages may not be accurate or reliable. Further, it is difficult todetermine the exact penetration depth of gastric cancer. Although NBIimages use different wavelengths of light to penetrate and thusvisualize different layers and tissues (e.g., blood vessels) within thestomach wall, the precise depth of the cancer cannot be determinedbecause NBI images show, for example, blood vessels present not only atthe depth at which the wavelength band of light can reach, but also showblood vessels present along the penetration route of the light and inthe vicinity of the depth at which the wavelength band of light canreach. Thus, the exact depth of the cancer cannot be determined from NBIimages.

The present systems and methods are designed to improve the accuracy andconsistency of gastric cancer diagnoses, and enable the exact depth ofcancer to be determined. FIGS. 26A and 26B show exemplaryalgorithms/methods 1000, 1014 for diagnosing a subject with gastriccancer and a particular severity (e.g. stage) thereof according thepresent disclosure. The method 1000 of FIG. 26A includes a first step1002 of capturing endoscopic images of a subject's stomach includingimages of different wavelength bands to visualize different layers andtissues within the stomach wall. Difference images are then generatedfrom the subject images in step 1004 for improved visualization ofparticular layers and/or tissues within the stomach wall. In the method1000 of FIG. 26A, the subject images are then compared to correspondingreference images representative of particular severities (e.g., stages)of gastric cancer to identify reference images with similar featurepatterns in step 1006. Multiple subject images, including differenceimages, may be compared with corresponding reference images to determinethe severity or degree of cancer present in each layer of the stomachwall and the precise penetration depth of cancer. The determinations ofthe severity of cancer present in each layer of the stomach wall canthen be compiled and analyzed for facilitating an accurate diagnosis ofan overall severity level (e.g., stage) of gastric cancer in step 1010.The diagnosis may be made based not only on the comparison in step 1006but also on clinical data 1008. A report of the diagnosis may be outputby the system in step 1012.

FIG. 26B shows another method/algorithm 1014 for diagnosing gastriccancer and a severity level (e.g., stage) thereof in a subject. Themethod/algorithm 1014 of FIG. 26B is substantially the same as themethod/algorithm 1000 in FIG. 26A except that step 1016 is performedinstead of step 1006. Therefore, subject images are obtained in step1002, and difference images are generated in step 1004. But instead ofcomparing the subject images to reference images indicative of aparticular severity (e.g., stage) of gastric cancer, the subject images,including the wavelength and difference images, are input into anArtificial Intelligence (AI) model that has been trained based on thereference images to diagnose the subject with gastric cancer and a stagethereof. The subject can then be diagnosed in step 1010, and a reportmay be output in step 1012. The diagnosis may be made based not only onthe out of the learned model but also other clinical data 1008.

In both methods 1000, 1014, the first step 1002 involves acquiring anendoscopic image of a subject's stomach by capturing the reflected lightwhen the tissue is illuminated by light. Images of different tissue(e.g., blood vessels) and/or layers of the stomach wall can be obtainedby using light of different wavelength bands because a depth of lightpenetrating tissue varies according to a wavelength of the light. Forexample, as illustrated in FIG. 24, blue 986, green 984, or infraredlight 982 light penetrates different layers of the stomach wall, andthus, an image of the mucosa 972, submucosa 974, and muscle 976 may beobtained according to a wavelength band of the illumination light.

The degree to which light is scattered by living tissue can also varydepending on the wavelength band of light. The refractive index of thetissue can also affect the degree to which light is scattered by theliving tissue. In particular, in the case of light with a shortwavelength band, such as blue light 986, the light only reaches aroundthe surface layer due to the absorption properties and scatteringproperties at the living body tissue, being subjected to absorption andscattering within the range up to that depth, so light coming out fromthe surface is observed. In the case of green light 984 with awavelength band longer than that of blue light 986, the light reaches adepth deeper than the range where the blue light 986 reaches, issubjected to absorption and scattering within the range at that depth,and light coming out from the surface is observed. Further, red lightwith a wavelength band longer than that of green light 984, reaches arange even deeper. Infrared (IR) light 982 with an even longerwavelength band reaches a range even deeper.

Although not illustrated in FIG. 24, various other wavelength bands oflight, such as red, violet, ultraviolet light, blue-green, violet-blue,white, and the like can be used for penetrating different layers of thestomach and/or imaging different tissue and/or structures (e.g., bloodvessels and lesions) within the layers. As used herein, “wavelengthimage” refers to an image obtained by a particular wavelength band oflight, including, but not limited to, ultraviolet, violet, violet-blue,blue, blue-green, green, red, near-infrared, infrared, and white lightimages.

The wavelength images may be acquired by controlling the wavelength bandof the light illuminated during imaging to be a desired wavelength band.As one example, a narrow-band imaging (NBI) endoscope can be used toobtain the wavelength images. An NBI endoscope can separate visiblelight of a wide band into, for example, blue, green, or red light ofnarrow bands using a rotary filter wheel. The separated narrow bandlight is sequentially or selectively illuminated onto a particular partof the stomach to obtain an image. Wavelength images may include imagesobtained with white, violet, violet-blue, blue, blue-green, green, red,ultraviolet, infrared, and near-infrared light. The wavelength imagesmay be obtained by illuminating light of a particular wavelength bandand capturing the reflected light. Alternatively, wavelength images (ofa narrower wavelength band than white light) can be obtained from whitelight images by decomposing the images into wavelength components, andsynthesizing images of desired wavelength bands. An arbitrary spectralimage may be extracted from an image acquired with white light (400 to700 nm) as in Flexible Spectral Imaging Color Enhancement (FICE) toobtain a narrow wavelength band image.

A lamp may be used as the light source to illuminate light having aparticular wavelength band, or a laser may be used as the light source.In the former case, the lamp light of a particular wavelength bandreaches a depth having a width in the depth direction of the stomachwall. In the latter case, the wavelength band of the laser is verynarrow. Therefore, the depth to which the laser reaches does not have awidth. As such, it is possible to determine whether or not the cancerhas reached a specific depth (e.g., 1 mm in depth, 2 mm in depth, etc. .. . ) by using a laser. For example, ultraviolet light can be used todetect/diagnose very early cancers occurring in the surface layer of themucosal layer. Near-infrared light can also be used to detect anddiagnose cancers that develop in the muscularis (deeper layers of thestomach wall). Table 3 shows the invasion depth of exemplary lamp andlaser lights of various wavelength bands.

TABLE 3 Light source Name Wavelength band Invasion depth Lamp White340-700 nm Muscular layer Purple color 265-310 nm Mucosal layer Blue390-445 nm Mucosal layer Green 530-550 nm Submucosal layer Infrared790-820 nm, Muscular layer 905-970 nm Laser Solid Ho:YAG 206 nm Mucosallayer state Er:YAG 294 nm Mucosal layer Ruby 694 nm Muscular layer Nd:glass 1060 nm Muscular layer Nd:YAG 1061 nm Muscular layer Gas Excimer193-353 nm Mucosal layer state He-Cd 325 nm (442 nm) Mucosal layer Ne332 nm Mucosal layer N2 337 nm Mucosal layer Xe 460-627 nm Submucosallayer Ar+ 458-515 nm Submucosal layer Kr 472-647 nm Submucosal layer Cu500 nm Submucosal layer CO 530 nm Submucosal layer He-Ne 633 nmSubmucosal layer (1150 nm, 3390 nm) CO2 928 nm Muscular layer (962 nm,1060 nm) Semi- InGaN 450 nm Mucosal layer conductor ZnCdSe 489 nmMucosal layer ZnTeSe 512 nm Submucosal layer GaP 555 nm Submucosal layerAGaInP 570 nm Submucosal layer InGaN 590 nm Submucosal layer AlGaAs 660nm Muscular layer GaP(Zn-0) 700 nm Muscular layer GaAs(Si) 980 nmMuscular layer InGaAsP 1300 nm Serosa

White light images can be used to visualize unevenness of the stomachwall surface and the color of the stomach wall surface (e.g., redness).Light (e.g., from a lamp or laser) of a narrower wavelength band can beilluminated to visualize specific layers of the stomach or specifictissues within the stomach wall layers. For instance, FIG. 24 showspenetration of various layers of the stomach wall by exemplarywavelength bands of light, e.g., blue 986, green 984, and infrared 982lights. Blue light (e.g., wavelength band of 390-445 nm) reaches themucosal layer 972, and green light (e.g., wavelength band of 530-550 nm)reaches the submucosal layer 974. Various images showing differentlayers of the stomach can be acquired by changing the wavelength band oflight. Additionally, different wavelength bands of light absorb/reflectdifferent tissues. Therefore, the structure that can be observed alsodiffers depending on the wavelength band. For example, white light(e.g., wavelength band of 340-700 nm) is absorbed/reflected byhemoglobin. Blue light (e.g., wavelength band of 390-445 nm) delineatesblood vessels containing hemoglobin.

Images acquired with violet light (e.g., wavelength band of 265-310 nm)can provide vascular image information of the surface layer of themucosal layer. Images acquired with blue light (e.g., wavelength band of390-445 nm) can provide information on the vascular image of the mucosallayer. Images acquired with green light (e.g., wavelength band of530-550 nm) can provide information on the blood vessel image of thesubmucosa. An image obtained by near-infrared light (e.g., a wavelengthband of 905 to 970 nm) can provide information on a blood vessel imageof a muscular layer.

Lesion or tumor areas may be identified from the microvascular pattern.Gastric cancer builds up blood vessels to share nutrients with cancercells. Therefore, gastric cancer, including early gastric cancer, may bedetected and diagnosed by analyzing the microvascular pattern andmicrosurface structures of the superficial mucosa. For instance, alesion area can be determined as an area in which blood vessels aredensely present. A regular microvascular pattern may be one in which inwhich the mucosal capillaries have a uniform shape that can beclosed-loop (polygonal) or open-loop, and a consistent size, and theirarrangement and distribution are regular and symmetrical. An irregularmicrovascular pattern, on the other hand, may be one in which thevessels differ in shape, are closed-loop (polygonal), open-loop,tortuous, branched, bizarrely shaped, with or without a network. In anirregular microvascular pattern, the size of the vessels may also varyand their arrangement and distribution may be irregular andasymmetrical. For instance, an irregular microvascular pattern may bedefined by the presence of thin spiral blood vessels within the finelobular superficial structure, or the presence of vertical spiral bloodvessels within the coarse lobular superficial structure. Other irregularmicrovascular patterns may include fine networks including fine tubularstructures surrounded by thin microvasculature or corkscrew patterns,which appear as obliterated surface structures and irregular vascularpatterns without loop formation. A microvascular pattern may be absentwhen the subepithelial microvascular pattern is obscured by the presenceof an opaque substance, for example, white opaque substance, within thesuperficial part of the mucosa. An irregular or absent microvascularpattern may be indicative of gastric cancer.

In step 1004 of FIGS. 26A and 26B, difference images can be generated tovisualize particular layers of the stomach wall, and/or particulartissues or structures (e.g., blood vessels, lesions) within the layers.As used herein, “difference image(s)” refers to subtraction images,addition images, and combinations thereof. For example, a differenceimage may be generated by comparing or adding a first image and a secondimage. One or both of the first and second images may be wavelengthimages of different wavelength bands of light, or may be differenceimages. For example, a difference image may be generated by subtractinga second wavelength image of a second wavelength band of light from afirst wavelength image of a first wavelength band of light, longer thanthe second wavelength band, or a difference image may be generated bysubtracting a wavelength image of a specific wavelength band of lightfrom a previously generated difference image. A difference image mayalso include addition images obtained by adding a first image to asecond image. One or both of the first and second images may bewavelength images of different wavelength bands of light, or may bedifference images. Additionally, a difference image may include anaddition/subtraction image obtained by adding a first image to a secondimage and subtracting a third image from the addition image. Forexample, such a difference image may obtained by adding a firstwavelength image to a second wavelength image (e.g., white light image)and subtracting a second wavelength image from the addition image.Different layers or combinations of layers can be visualized by changingthe images being subtracted and/or added.

Table 4 below shows exemplary wavelength bands of the first and secondwavelength images for acquiring images of specific stomach layers.Although Table 4 only shows exemplary stomach layers images by obtainingsubtraction-type difference images, addition images in which a firstimage and a second image of a different wavelength band are addedtogether, as well as an addition/subtraction image in which a firstwavelength image is added to a second wavelength image (e.g., a whitelight image), and then a third wavelength image is subtracted, can alsobe obtained for facilitating accurate diagnosis of gastric cancerseverity.

TABLE 4 Stomach Layer Visualized in First Image Second Image* DifferenceImage (wavelength) (wavelength) Upper Mucosa Blue None 972a (440-460 nm)Mucosa 972 Green None (upper 972a and (530-550 nm) lower 972b) LowerMucosa Blue (440-460 nm) 972b Mucosa 972, Infrared None Submucosa 974,(940 nm) and muscle layer 976 Lower Mucosa Blue (440-460 nm) 972b,Submucosa 974, and muscle layer 976 Submucosa 974, Green (530-550 nm)and muscle layer 976 Mucosa 972 White None (upper 972a and (340-700 nm)lower 972b), Submucosa 974, and upper part of muscle layer 976 Upperpart of Blue (440-460 nm) muscle layer 976 Submucosa 974, Green (530-550nm) and upper part of muscle layer 976 *The Second Image is subtractedfrom the First Image to create the Difference Image.

The subject images include wavelength, white light, difference,addition, and addition/difference images. For example, the subjectimages may include one or more of the following types of images: (1)wavelength images obtained using lamp or laser light source including(a) a white light image with information on the surface irregularitiesand color of the stomach wall, (b) a narrower wavelength band image withinformation on tissue(s) and/or structure(s) in the various stomach walllayers, and (2) a difference image obtained by: (a) subtracting a secondwavelength image of a second wavelength band from a first wavelengthimage of a first wavelength band (e.g., the first wavelength image maybe a white light image or a narrower wavelength band image), (b) anaddition image obtained by adding a first wavelength image to a secondwavelength light image (e.g., the first or second wavelength images maybe a white light image or a narrower wavelength band image), or (c) anaddition/subtraction image obtained by adding a first wavelength imageto a second wavelength light image and subtracting a third wavelengthimage from the addition image (e.g., the first or second wavelengthimages may be a white light image or a narrower wavelength band image).The images may include those acquired with lamp light of a particularwavelength band and those acquired with a laser of a narrower wavelengthband. The more subject images obtained and compared to reference images,the more accurate the diagnosis of the severity (e.g., stage) of gastriccancer.

In step 1006 of method 1000 of FIG. 26A, the subject images, includingthe wavelength and difference images, are compared to reference imageindicative of a particular severity level (e.g., stage) of gastriccancer. The reference images also include images representative of anormal or healthy stomach that does not have gastric cancer. The subjectimages are compared with corresponding reference wavelength and/ordifference images to identify to identify reference images with similarfeature patterns. The feature patterns may include microvascularpatterns and the size and depth of lesions. The subject images arecompared to corresponding reference images to diagnose the subject witha particular severity level (e.g., stage) of gastric cancer. Forexample, a blue wavelength subject image is compared with a bluewavelength reference image in the database, or a subject differenceimage of green wavelength and blue wavelength subject images is comparedwith a reference difference image of green wavelength and bluewavelength reference images. Accuracy can be improved by acquiringmultiple images of the subject showing various stomach layers andtissues, and comparing each of those images with corresponding referenceimages in the database. For example, by acquiring multiple images, thedifferent layers and tissues in the stomach wall can be individuallyexamined to determine the extent or degree of gastric cancer for eachstomach wall layer. This facilitates determining a depth of the cancerin the stomach wall for making an accurate diagnosis. An overalldiagnosis of the severity (e.g., stage) of gastric cancer can be madebased on the individual determinations for each stomach wall layer.

The use of multiple images including images of a particular wavelengthband (“wavelength images”) and difference images provides significantlyimproved visualization of the various layers of the stomach wall, aswell as various tissues and structures within the stomach wall layers,enabling the location, size (e.g., lateral extent), and invasion depthof a tumor or lesion to be readily determined. The difference imagesenable feature patterns corresponding to the severity of gastric cancer,such as a malformation, lesion, tumor, and/or microvascular pattern,including thickness and branching of blood vessels, the occupation rate,and the number of blood vessels per area, to be clearly visualized. Atopography and color (e.g., redness) of the stomach wall surface canalso be observed when one of the wavelength images is a white lightimage. Additionally, by processing and comparing the images as discussedherein, lesions or tumors can be identified and analyzed to makeaccurate and consistent diagnoses of the severity and/or stage ofgastric cancer. For instance, by comparing subject images to referenceimages indicative of various stages of gastric cancer, feature patterns,such as lesions, tumors, and irregular microvascular patterns, can bemore reliably detected. Accuracy and consistency of diagnoses canfurther increase as the databases of reference images increases.

By acquiring a plurality of images of the subject's stomach, includingwavelength and difference images, each layer of the subject's stomachand the surface topography and color can be visualized. Feature patternscan be extracted from the images to determine how far into the stomachlayers the cancer has spread. For example, how far the cancer haspenetrated into each of the stomach layers can be determined byacquiring images of the upper layers 972 a and lower layers 972 b of themucosa 972, and the submucosa 974, individually, and determining theextent or severity of gastric cancer in each layer. Then, thedeterminations for each layer can be compiled and analyzed to make anoverall diagnosis of the severity (e.g., stage) of gastric cancer.Lesion(s) 998 that are present in the submucosal layer 974 or musclelayer 976, but are not present in the mucosal layer 972, which areindicative of Scirrhous stomach cancer, may be identified by acquiringand visualizing multiple images (including wavelength and differenceimages). For example, a difference image of an infrared (first)wavelength image and a green (second) wavelength image would allowvisualization of the submucosa 974, and a green wavelength image wouldallow visualization of the mucosa 972. If feature patterns of cancer arepresent in the submucosa 974 but not in the mucosa 972, then the subjectcould be diagnosed with Scirrhous stomach cancer 998.

The comparison in step 1006 is then used to make a diagnosis in step1010. The diagnosis 1010 may also be made by taking other data andanalysis 1008 into consideration, including non-image data, such asfamily history, clinical data, laboratory data, subject history, familyhistory, subject vital signs, results of various tests (e.g., genetictests), and any other relevant non-image data. Numerous subject andreference data and images may be created and compared. Then, a report ofthe diagnosis is output in step 1012, for example, to an output device18 (shown in FIG. 1), such as a display or a printer, or to a databaseor server for storage, or to a user in a human-readable format.

In the method 1014 of FIG. 26B, the subject images, including thewavelength and difference images, are input into an AI model that hasbeen trained based on the reference images to diagnose the subject withgastric cancer and a stage thereof instead of comparing the subjectimages to reference images indicative of a particular severity (e.g.,stage) of gastric cancer. For example, the AI model may be a deeplearning model, such as convolutional neural network. The referenceimages are labeled as representing a healthy stomach free of gastriccancer, or labeled as representing a stomach having a particularseverity (e.g., stage) of gastric cancer. The gastric cancer diagnosisand severity (e.g., stage) assigned to each of the reference images maybe based on the results of diagnostic imaging or tissue diagnosis.Severity labeling may begin with imaging or histological diagnosis by adoctor and metastasis to lymphatic vessels or other organs by CT and/orMRI. During training, e.g., supervised learning, the model learnsfeature patterns corresponding to gastric cancer and a severity thereoffrom the reference images collected in the database. The model may betrained via a pattern recognition algorithm to extract and learn thefeature patterns corresponding to healthy and various severity levels(e.g., stages) of gastric cancer. For example, such feature patterns mayinclude microvascular patterns, such as blood vessels and feature pointsbased on the thickness, branching, and position of bloods vessels andthe number of blood vessels per area, occupation rate, topography (e.g.,unevenness) of surface of stomach wall and color (e.g., redness), andmalformations.

The system may perform data analytics to determine meaningful patternsin image and non-image data and build models based on these determinedpatterns, which can be used to automatically analyze images and othermedical data. For example, after developing a model using training data,the system may update the model based on feedback designating acorrectness of the training information or a portion thereof. Forexample, the system may update a model based on clinical resultsassociated with one or more images included in the training information.In some embodiments, a user may manually indicate whether diagnosticinformation included in the training information was correct as comparedto an additional (e.g., later established diagnosis).

FIG. 29 shows an exemplary method/algorithm 1100 for preparing (e.g.,processing) the reference images and training the model using thereference images. The reference images and associated diagnostic results(presence or absence of stomach cancer and severity (stage)) arecollected and stored in a database. First, the reference images arerandomly classified into two groups in step 1102. The first group is thelearning data set, which classifies about two thirds of the total data.The second group is a data set for validation (e.g., verification), andapproximately one third of all data is classified. Then in step 1104,the images are read and images belonging to the following exclusioncriteria are excluded. Excluded images include subjects with severegastritis, gastric ulcers or lesions adjacent to ulcers, poor qualityimages, bleeding, halation, blurring, and defocus. After exclusion,images of approximately 800 patients are used. The learning data setuses about 8,000 images. The images are stored in JPEG form and used fortraining. The reference images can be rotated, expanded/reduced, moved,or distorted to increase the number of pieces of data that can be usedto train the AI model.

The reference images are then vectorized in step 1106. Specifically, awhite image and a difference image are vectorized, respectively, and thevectorized white image and the difference image are integrated into onepiece of data. Next, the diagnosis result (presence or absence ofgastric cancer and severity (stage)) is associated with the vectorizedreference image in step 1108. Vectorization and related of diagnosticresults are performed for all reference images. The vectorization ofimages may be performed by algorithm, such as Potrace. The algorithm isbasically vectorized by 1) extraction of contour coordinates, 2)polygonization, and 3) approximation with a Bezier curve.

Then in step 1110, the learned model is then trained using the firstgroup of data (e.g., training dataset). When creating the learned model,the number of nodes is appropriately changed in accordance with theoutput accuracy of the learned model to create the learned model.Thereafter, in step 1112, the reference images classified into thesecond group (e.g., validation dataset) are input to the learned model,and a diagnosis result is output. The matching rate between the resultoutput by the learned model and the diagnostic result associated withthe reference image is calculated to determine the accuracy of thelearning model.

Further, the first group and the second group create a plurality oflearned models by changing the combination of reference images. In step1114, the learned model with the highest precision is selected among theplurality of learned models created and used to analyze the subjectimages in the methods and systems disclosed herein for diagnosing aseverity (e.g., stage) of gastric cancer.

The learned model with the highest precision is employed in the systemsand methods disclosed herein. When a subject image is input into thelearned model, the learned model can identify the presence and severityof gastric cancer from the subject image based on the learned featurepatterns from training on the plurality of reference images in thedatabase. The learned model extracts features from the subject imagesand outputs a severity (e.g., stage) diagnosis based on the featurepatterns. Accuracy and consistency of diagnoses can be improved by theuse of AI to extract feature patterns related to the severity of gastriccancer using a model trained with multiple reference images.Additionally, accuracy is improved when multiple subject images (e.g.,including various white light, narrower wavelength, and differenceimages) such that the learned model can identify the severity of cancerpresent in each layer of the stomach wall to determine a penetrationdepth of gastric cancer in the stomach wall and diagnose a stage ofgastric cancer.

In addition to subject images, the learned model may also receivenon-image data for analysis and may diagnose the subject as havinggastric cancer and/or a severity or stage thereof based on both theimage and non-image data. For example, the subject's images and medicalrecord information may be input into the system. The system can diagnosegastric cancer based on learned information and the images using themodel. The system may identify the severity (e.g., stage) of gastriccancer based on the above diagnosis and medical record information(e.g., presence or absence of cancer metastasis). The system may outputto a user a diagnosis of the presence or absence of gastric cancer andthe stage of gastric cancer. As more and more subjects are diagnosedwith various severity levels and stages of gastric cancer, the systemmay update its models accordingly. For example, the learned model may bedesigned to adjust itself in response to new data and conditions. Forexample, the model may be retained with new subject images after theyare diagnosed as showing a particular severity level (e.g., stage) ofgastric cancer.

The learned AI model can then output a diagnosis in step 1010 based onthe subject images. The diagnosis may also be made by taking other dataand analysis 1008 into consideration, including non-image data, such asfamily history, clinical data, laboratory data, subject history, familyhistory, subject vital signs, results of various tests (e.g., genetictests), and any other relevant non-image data. Numerous subject andreference data and images may be created and compared. Then, in step1012, a report of the diagnosis is output, for example, to an outputdevice 18 (shown in FIG. 1), such as a display or a printer, or to adatabase or server for storage, or to a user in a human-readable format.

The various images and data described herein may be stored in one ormore databases to facilitate subsequent data analysis. Moreover, any orall of the comparisons may be performed either automatically by a dataprocessing system, such as system 10, or by a medical professional, suchas a doctor, or by some combination thereof, to facilitate automatic ormanual diagnosis of the subject.

The present embodiments may use a database of reference/training images(including wavelength and difference images) indicative of variousseverity levels (e.g., stages) of gastric cancer, and normal (healthy)stomach walls that do not have gastric cancer. The reference images maybe images collected from past patients, preferably of various ages,genders, and ethnicities. The database is constructed to includereference image data including wavelength images acquired at differentwavelength bands (including white light images) and difference images.The diagnosis result is associated with each reference image and otherdata in the database. Accuracy and consistency of diagnoses increase asthe number of reference images, as well as reference severityinformation, in the database increases. The database may be stored on aserver, in one or more memory or storage devices, and/or in othersuitable media. Such databases may be continuously or periodicallyupdated as more subjects are diagnosed with a particular stage ofgastric cancer.

Additionally, reference image data may be categorized and sorted intostandardized databases, such as through an exemplary method 1020 shownin FIG. 25. The method 1020 may include acquiring reference data 1022,which may include image and non-image data from various people, andcategorizing the data in step 1024. For example, the reference data 1022may be categorized into various groups, such as normal (healthy) subjectdata 1026, data 1028 of subjects clinically diagnosed with low severitygastric cancer, data 1030 of subjects diagnosed with moderate severitygastric cancer, and data 1032 of subjects diagnosed with high severitygastric cancer. The data 1026, 1028, 1030, and 1032 may be stored inrespective databases 1034, 1036, 1038, and 1040. Such databases may bestored on a server, in one or more memory or storage devices, and/or inother suitable media. Such databases may be continuously or periodicallyupdated as more subjects are diagnosed.

Each database for normal 1034, low severity 1036, moderate severity1038, and high severity 1040 gastric cancer may include one or more ofthe following types of reference images: (1) wavelength images obtainedusing a lamp or laser light source including (a) a white light imagewith information on the surface irregularities and color of the stomachwall, (b) a narrower wavelength band image with information on tissue(s)and/or structure(s) in the various stomach wall layers, and (2) adifference image obtained by: (a) subtracting a second wavelength imageof a second wavelength band from a first wavelength image of a firstwavelength band (e.g., the first wavelength image may be a white lightimage or a narrower wavelength band image), (b) an addition imageobtained by adding a first wavelength image to a second wavelength lightimage (e.g., the first or second wavelength images may be a white lightimage or a narrower wavelength band image), or (c) anaddition/subtraction image obtained by adding a first wavelength imageto a second wavelength light image and subtracting a third wavelengthimage from the addition image (e.g., the first or second wavelengthimages may be a white light image or a narrower wavelength band image).The images may include those acquired with lamp light of a particularwavelength band and those acquired with a laser of a narrower wavelengthband. Multiple images (e.g., multiple different wavelength images, whichare used to produce multiple different difference images) may be storedfor each stage of gastric cancer, each layer of the stomach wall, and/oreach tissue within the stomach wall. The reference images areaccumulated in a database together with the respective stage of gastriccancer or may be identified as “normal” or “healthy” for images ofstomachs that do not have gastric cancer. The data 1026, 1028, 1030, and1032 in each database 1034, 1036, 1038, and 1040 may be furtherstandardized and classified according to various subjectcharacteristics, such as age, gender, and race, or type of image, suchas white light images, blue wavelength images, purple wavelength images,red wavelength images, green wavelength images, ultraviolet wavelengthimages, infrared wavelength images, near-infrared wavelength images, anddifference images generated from various wavelength images.

An exemplary embodiment is shown in FIG. 27A, in which a subject isdiagnosed with severity 2 gastric cancer (which may correspond to stageII gastric cancer). One or more wavelength images of the subject areacquired and difference images are generated for the purpose ofvisualizing different stomach layers. The wavelength images and/ordifference images are compared with corresponding reference wavelengthand/or difference images indicative of various severity levels ofgastric cancer in the database to identify similar reference images inorder to diagnose the subject with a particular severity level (e.g.,stage) of gastric cancer. Although FIG. 27A only shows comparingdifference images, wavelength images and white light images that havenot been processed as a difference image can also be used in thecomparison. Similarly, the subject images are compared to multiplereference images indicative of various severities of gastric cancer, notjust the relevant severity 2 (e.g., stage II) reference images shown inFIG. 27A.

Then, the subject images are compared to a plurality of correspondingreference images in the database to identify reference images withsimilar feature patterns, and determine the severity of gastric cancerof the subject. The feature patterns may include microvascular patternsand the size and depth of lesions. By acquiring multiple images,different layers of the stomach can be imaged for determining how deepthe cancer has invaded into the stomach in order to facilitate making anaccurate diagnosis of the severity. In FIG. 27A, feature patterns, suchas blood vessels and feature points based on the thickness of bloodsvessels and position of the blood vessels/branches, have been extractedfrom the images to facilitate the comparison between the subject imagesand the reference images for making an accurate diagnosis of theseverity.

Alternatively, the database shown in FIG. 27A may be an AI database inwhich multiple reference images indicative of various severities (e.g.,stages) of gastric cancer have been accumulated. The reference imagescollected in the database are used to train an AI model, such as a deeplearning model, e.g., a convolutional neural network. The referenceimages are labeled as representing a particular severity (e.g., stage)of gastric cancer or as representing a normal or healthy stomach free ofgastric cancer. Gastric cancer diagnosis and severity may be assigned tothe reference images based on the results of diagnostic imaging ortissue diagnosis. Severity labeling begins with imaging or histologicaldiagnosis by a doctor and metastasis to lymphatic vessels or otherorgans by CT and/or MRI. The model learns feature patterns correspondingto gastric cancer and a severity thereof from the reference imagescollected in the database. For example, such feature patterns mayinclude microvascular patterns, such as blood vessels and feature pointsbased on the thickness, branching, and position of bloods vessels andthe number of blood vessels per area, occupation rate, topography (e.g.,unevenness) of surface of stomach wall and color (e.g., redness), andmalformations. Once the model has been sufficiently trained, a learnedmodel is created.

When a subject image is input into the learned model, the learned modelcan identify the severity of gastric cancer from the subject image basedon the learned feature patterns from training on the plurality ofreference images in the database. The learned model extracts featuresfrom the subject images and outputs a severity (e.g., stage) diagnosisbased on the feature patterns. Accuracy and consistency of diagnoses canbe improved by the use of AI to extract feature patterns related to theseverity of gastric cancer based on a training model of multiplereference images.

The diagnosis may also be made by taking other data and analysis intoconsideration, including non-image data, such as family history,clinical data, laboratory data, subject history, family history, subjectvital signs, results of various tests (e.g., genetic tests), and anyother relevant non-image data. The various images and data describedherein may be stored in one or more databases to facilitate subsequentdata analysis. Moreover, any or all of the comparisons may be performedeither automatically by a data processing system, such as system 10, orby a medical professional, such as a doctor, or by some combinationthereof, to facilitate automatic or manual diagnosis of the subject.

FIG. 27B shows an exemplary method in which the mucosal layer isexamined and diagnosed as showing Stage II gastric cancer using whitelight, blue light, and purple light images. White light, blue light, andpurple light images of a subject's stomach wall are obtained. An imageobtained by white light (e.g., wavelength band of 340-700 nm) can showunevenness or color of the surface of tissue. Blue light (e.g.,wavelength band of 390-445 nm) can reach the mucosal layer of thestomach wall and is further absorbed and/or reflected by hemoglobin.Blue light (e.g., wavelength band of 390-445 nm) can delineate bloodvessels in the superficial layer of the mucosal layer. The purple light(e.g., wavelength band of 265-310 nm) can reach the surface layer of thestomach wall (the surface layer of the mucosal layer), and is furtherabsorbed and/or reflected by hemoglobin. Purple light (e.g., wavelengthband of 265-310 nm) can delineate blood vessels in the superficial layerof the mucosal layer.

A difference image (difference b-v in FIG. 27B) is generated bysubtracting the purple light image from the blue light image. Thedifference image b-v is a blood vessel image of the mucosal layer (ablood vessel image obtained by removing the surface layer of the mucosallayer from the mucosal layer).

Further, a difference image (difference w-b-v in FIG. 27B) is created bysubtracting the difference image (difference b-v) from the white lightimage. The surface of the stomach wall contains digestive enzymes (e.g.,amylase, protease, lipase, maltase) and saline. Therefore, the surfaceof the stomach wall is highlighted by digestive enzymes andphysiological saline present on the surface of the gastric juice. As aresult, surface highlighting also occurs in acquired white light images.The surface highlighting can be deleted by subtracting the differenceimage (difference b-v) from the white light image.

In the same manner as described above, reference images (e.g., white 1,white 2, difference 1 w-b-v, difference 2 w-b-v, . . . ) indicative ofspecific severities (e.g., stages) of gastric cancer are stored in thedatabase. The reference images includes a white light image (white 1,white 2, . . . ) and difference images (difference 1 w-b-v, difference 2w-b-v, . . . ) generated from reference wavelength images. Further, thereference images (difference image, white light image) accumulated inthe database are each associated (e.g., labeled) with their diagnosisresult (e.g., presence or absence of stomach cancer, severity (stage)).Although the database in FIG. 27B only shows Stage I and Stage II imagesas exemplary reference images, the database preferably contains multipleimages representative of all the various stages of gastric cancer, aswell as images showing normal (healthy) stomachs that do not havegastric cancer.

Then, in FIG. 27B, diagnoses are performed by comparing the referenceimages (difference images, white light images) stored in the databasewith the subject wavelength image (e.g., white light image) anddifference image (e.g., difference w-b-v). Diagnosis includes anassessment of stomach cancer, reliability of gastric cancer assessment(A %), severity (stage) (e.g., Stage II), and reliability of stageassessment (a %).

“Reliability” refers to, for example, how many patients who are judgedto be positive for gastric cancer (output results) are confirmed to havegastric cancer by definitive diagnosis. Specifically, it can becalculated by the following equation.

$\frac{\text{True Positive}}{\text{True Positive} + \text{False Positive}} = \frac{\text{Positive is the number of correct answers}}{\text{Total number of positives}}$

Similarly, the reliability of stage assessment refers to how manypatients who are judged as having a particular stage of gastric cancerare confirmed to have that stage of gastric cancer.

As discussed above, AI (deep learning) may be used instead of comparingthe subject images to the references images in the database. In thiscase, the AI creates a learned model by learning the reference images(difference image, white light image in FIG. 27B) and the diagnosisresult (presence or absence of stomach cancer and severity (stage))associated with each image. Diagnostics are then performed by inputtinga difference w-b-v and a white light image of the subject to the learnedmodels. In the case of using AI, learning and diagnosis can be performedwith higher accuracy than in the case of learning and diagnosing onlywith a difference image or only with a white light image by learning anddiagnosing using a white light image and a difference image.

FIG. 27C sows an exemplary method in which the superficial gastric wallis diagnosed as showing Stage II gastric cancer using white light andpurple light images. White light and purple light images of a subject'sgastric wall are obtained. An image obtained by white light (e.g.,wavelength band of 340-700 nm) can show unevenness or color of thesurface of tissue. The purple light (e.g., wavelength band of 265-310nm) reaches the surface layer of the stomach wall (the surface layer ofthe mucosal layer), and is further absorbed and/or reflected byhemoglobin. That is, purple light (e.g., wavelength band of 265-310 nm)can delineate blood vessels in the superficial layer of the mucosallayer. Therefore, cancer reaching the surface layer of the stomach wallcan be diagnosed by an image acquired by purple light. A subjectdifference image (difference w-v) is created by subtracting the purplelight image from the white light image. The mucosal surface layer isdelineated by the difference between the white image and the purpleimage in the subject difference image (difference w-v). In addition, theinfluence of reflection by the body fluid of the image can be excludedby the difference image.

In the same manner as described above, reference images indicative ofparticular severities (e.g., stages) of gastric cancer (e.g., white 1,white 2, difference 1 w-v, difference 2 w-v, . . . ) are stored in thedatabase. The reference images includes a white light image (white 1,white 2, . . . ), and difference images (difference 1 w-v, difference 2w-v, . . . ) generated from reference wavelength images. Further, thereference images (difference image, white light image) accumulated inthe database are each associated (e.g., labeled) with their diagnosisresult (e.g., presence or absence of stomach cancer, severity (stage)).Although the database in FIG. 27C only shows Stage I and Stage II imagesas exemplary reference images, the database preferably contains multipleimages representative of all the various stages of gastric cancer, aswell as images showing normal (healthy) stomachs that do not havegastric cancer.

Then, in FIG. 27C, diagnoses are performed by comparing the referenceimages (difference images, white light images) stored in the databasewith the subject wavelength image (e.g., white light image) anddifference image (e.g., difference w-v). Diagnosis includes anassessment of stomach cancer, reliability of gastric cancer assessment(B %), severity (e.g., stage II), and reliability of stage assessment (b%).

As discussed above, AI (deep learning) may be used instead of comparingthe subject images to the references images in the database. In thiscase, the AI creates a learned model by learning the reference images(difference image, white light image in FIG. 27C) and the diagnosisresult (presence or absence of stomach cancer and severity (stage))associated with each image. Diagnostics are then performed by inputtinga difference w-v and a white light image of the subject to the learnedmodels. In the case of using AI, learning and diagnosis can be performedwith higher accuracy than in the case of learning and diagnosing onlywith a difference image or only with a white light image by learning anddiagnosing using a white light image and a difference image.

The severity or extent of gastric cancer present on the surface and inmultiple layers of the stomach wall may be determined by acquiringmultiple images. For example, a severity diagnosis and reliabilitydetermination (x %) may be made for each layer of the stomach wall(including the surface) by any of the above methods. The individualseverity diagnoses and reliability determinations for the variousstomach wall layers may then be compiled and analyzed to determine anoverall diagnosis of the stage of gastric cancer. For example, a depthof gastric cancer may be determined from the diagnoses for each layer ofthe stomach, which is an important feature for diagnosing the stage ofgastric cancer. Clinical information regarding metastases may be furthertaken into consideration to make an overall gastric cancer stagediagnosis.

Alternatively, an ensemble learning AI model may be used to make anoverall stage diagnosis based on the diagnoses and reliabilitydeterminations for each stomach wall layer (including the surface). FIG.28A shows an exemplary algorithm for diagnosing the severity (e.g.,stage) of gastric cancer based on the severity of gastric cancer in eachstomach wall layer. For example, the severity of gastric cancer inindividual layers of the stomach wall, such as the superficial gastricwall (see, e.g., FIG. 27B), the muscosal layer (see, e.g., FIG. 27A),the submucosal layer, the muscle layer, and the like, is firstdetermined. The probability (e.g., C %, D %, E %) of cancer on thesurface, in the mucosal layer, submucosal layer, muscle layer, and thelike and the reliability (e.g., c %, d %, e %) of the diagnoses areoutput (output 1 in FIG. 28A). The outputs for the individual layers arethen input into an ensemble learning AI model to diagnose the subject aseither having or not having gastric cancer, and diagnose an overallgastric cancer severity (e.g., stage II) (output 2 in FIG. 28A). Thereliability (F %) of the diagnosis of gastric cancer and the reliability(f %) of stage assessment are also output.

FIG. 28B shows an exemplary algorithm for diagnosing the severity (e.g.,stage) of gastric cancer based on the severity of gastric cancer atspecific depths within the stomach wall (e.g., 1 mm in depth, 2 mm indepth, . . . ). The severity of gastric cancer at specific depths in thestomach wall can be determined by obtaining images using a laser as thelight source, which has a narrower wavelength band than lamp light. Forexample, ultraviolet light can be used to detect/diagnose very earlycancers occurring in the surface layer of the mucosal layer.Near-infrared light can also be used to detect and diagnose cancers thatdevelop in the muscularis (deeper layers of the stomach wall). Bydetermining the precise depth of gastric cancer in the stomach wall, theoverall stage of gastric cancer can be accurately diagnosed.

For example, the following steps may be employed to make an overalldiagnosis:

(A) Using AI specialized for a white light image and each differenceimage, whether or not cancer exists at a specific depth is output as a“probability of cancer existing in n-layer” (diagnostic result: stomachcancer probability: G %) and “reliability” (reliability: g %).

(B) Step (A) is performed for a plurality of depths to output theprobability (e.g., H % probability at depth of 2 mm) and reliability ofthe presence of cancer for various depths in the stomach wall (e.g., h %reliability of probability determination for 2 mm depth).

(C) The plurality of results output in (B) (“the existence probabilityof the cancer of X layer,” “the reliability”) is output to the AI modeltrained by ensemble learning, the overall severity (stage) and thereliability are output from a comprehensive viewpoint.

In FIGS. 28A and 28B, each AI (AI(a), AI(b) . . . ) outputs the presenceprobability and reliability of cancer for each layer or depth of thestomach wall. The output result is input to an ensemble learner (AI). Inthe AI (ensemble learner), training is performed by learning withteaching. Two types of information may be used for training. The firstis the probability and reliability of cancer from each AI (AI(a), AI(b). . . ) to each layer or depth of the stomach wall. The second is thediagnostic result associated with the reference images. The AI (ensemblelearner) weighs the diagnosis result associated with the reference imagewith the highest coincidence probability by repeatedly attempting whilechanging the weighting for the input (the existence probability andreliability of cancer for each depth or each layer of the stomach wall)from each AI (AI(a), AI(b), . . . ).

The present systems and methods enable determining the size and/orinvasion depth of the cancer into the stomach wall, which can be usefulfor determining appropriate treatment. For example, the subject imagesmay show relatively small size lesion(s) growing, for example, on thetop layer of cells of the mucosa, indicative of low severity gastriccancer. The subject images may show medium sized lesion(s) in the mucosaand/or submucosa, indicative of moderate severity gastric cancer. Thesubject images may show one or more large lesions in at least the mucosaand submucosa, indicative of high severity gastric cancer.

Determining the size of the lesion by the present systems and methodsnot only allows the severity of gastric cancer to be accuratelydiagnosed, but it also permits an appropriate method of treatment beemployed. In some embodiments, the system may also output informationtreatment recommendations based on the diagnosed severity (e.g., stage)of gastric cancer and size and depth of the cancer in the stomach wall.Table 5 shows a method of treatment indicated for each severity level ofgastric cancer and lesion size.

TABLE 5 Lesion Gastric Cancer Method of Diameter (d) Severity* treatmentd ≤ 5 mm 1 Hot biopsy 5 mm < d ≤ 20 mm 2 EMR** 20 mm < d ≤ 30 mm  3ESD*** *1 = low severity; 2 = moderate severity; and 3 = high severity.**EMR = Endoscopic Mucosal Resection ***ESD = Endoscopic SubmucosalDissection

The present methods may include treating the subject based on thedetermined severity level of gastric cancer. As shown in Table 5,different treatment methods are indicated for different sizelesions/different severity levels of gastric cancer. A brief discussionof the various treatment methods follows.

Hot biopsy may be indicated for low severity (1) gastric cancer lesions.For example, hot biopsy may be indicated for micro polyp removal (e.g.,having a diameter of 5 mm or less) and can be performed well in thelarge intestine with many small polyps. Hot biopsy involves removinglesion tissue while supplying high frequency current. Hot biopsy forcepscan be used to achieve hemostasis simultaneously with tissue collection.The procedure involves pulling the grasp tissue with the forceps suchthat the root of the lesion is thin and stretched, and passing highfrequency current in this state. The current is concentrated in thestretched tissue area, and the tissue is ablated and whitened. Afterconfirmation of tissue whitening, the tissue is torn and removed forcollection.

Endoscopic Mucosal Resection (EMR) may be indicated for moderateseverity (2) gastric cancer lesions. For instance, EMR may be indicatedfor removal of lesions having a diameter larger than 5 mm. For example,the lesion may have a size in a range of from 5 to 20 mm. EMR mayinvolve a submucosal injection under the lesion, snaring, and removingthe lesion. For example, physiological saline may be injected into thesubmucosal layer of a flat lesion to make it bulge. Then, the root ofthe raised lesion may be surrounded by a high frequency snare. Whileraising the snare, the snare is tightened (e.g., squeezed) around theroot of the raised lesion and high frequency current is applied to thelesion to cauterize and excise the lesion tissue.

Endoscopic Submucosal Dissection (ESD) may be indicated for highseverity (3) gastric cancer lesions. For example, ESD may be indicatedfor removal of lesions having a diameter larger than 5 mm or even largerthan 20 mm. ESD may be used to remove lesions up to about 30 mm or more.ESD allows en bloc resection of larger lesions. ESD may involvesubmucosal injection, circumferential mucosal precutting and dissection.For example, ESD may involve marking around the lesion, injectingphysiological saline into the submucosal layer of the lesion (e.g., toraise the lesion), incising the perimeter of the lesion, and removingthe incised submucosa layer to remove the entire lesion.

As mentioned above, gastric cancer stage may be determined based on thepenetration depth of gastric cancer in the layers of the stomach andmetastasis information. Subject images (e.g., wavelength and differenceimages) may be acquired for each layer of the stomach to determinewhether the cancer is present in the each of the layers of the stomach,such as the gastric wall surface, the mucosal layer, submucosal layer,and muscle layer. The images may images of various bands of light,including, for example, blue light, green light, red light, white light,and infrared bands of light, as well as difference images obtained bysubtracting various images from one another, adding various imagestogether, and combinations thereof, such as addition/subtraction imagesin which one image is added to another and then a third image issubjected from the addition image.

For example, with reference to FIG. 24, the subject wavelength imagesmay show cancer cells 988 only in the top layer of cells of the mucosa972, indicative of stage 0 gastric cancer. The subject wavelength imagesmay show lesion(s) 990 that have grown from the top layer of cells ofthe mucosa 972 into the next layers below, such as the lamina propriaand the muscularis mucosa, which is indicative of stage I gastriccancer. The subject wavelength images may show lesion(s) 992 that havegrown from the mucosa 972 into the submucosa 974, indicative of stage IIgastric cancer. The subject wavelength images may show lesion(s) 994that have grown from the mucosa 972 to the muscle layer 976, indicativeof stage III gastric cancer. The subject wavelength images may showlesion(s) 996 that have grown from the mucosa 972 deep into the musclelayer 976, indicative of stage IV gastric cancer. Finally, the subjectwavelength image may show lesion(s) 998 that are present in thesubmucosal layer 974 or muscle layer 976, but are not present in themucosal layer 972, which is indicative of Scirrhous stomach cancer.Therefore, the subject may be diagnosed with stage 0-IV gastric canceror Scirrhous stomach cancer based on a depth of lesion(s) in the stomachlayers. The methods and systems may also use electronic medical recordinformation, regarding, for example, whether the cancer has metastasizedto facilitate the diagnosis of the severity (e.g., stage) of gastriccancer, in combination with, for example, the depth and/or size oflesions in the subject stomach.

In particular, the diagnoses of gastric cancer stages may be furtherfacilitated by non-image data, such as information concerning whetherthe cancer has spread to nearby lymph nodes or whether the cancer hasspread to distant sites (metastasis). For example, such metastasisinformation in combination with the above gastric cancer depth and sizeinformation can be used to precisely diagnose the stage of gastriccancer. For example, if the tumor has grown from the top layer of cellsof the mucosa 972 into the next layers below such as the lamina propria,the muscularis mucosa, or submucosa 974, but has not spread to nearbylymph nodes or to distant sites, then the subject may be diagnosed withstage IA gastric cancer. On the other hand, if the tumor is the samesize, but the cancer has spread to 1 to 2 nearby lymph nodes (and hasnot spread to distant sites), then the subject may be diagnosed withstage IIB gastric cancer.

Thus, the subject images may be used to determine the size and/orinvasion depth of gastric cancer in the stomach layers to accuratediagnose the severity (e.g., stage) of gastric cancer. The presentsystems can further take into account information regarding metastasisof the cancer for accurately diagnosing the stage of gastric cancer.

Reference data, including image data and non-image data, may becollected from people or groups of people. Such people may includehealthy people that are not suffering from gastric cancer, and otherpeople suffering from different severity levels of gastric cancer. Thereference image and non-image data may be standardized and categorizedaccording to one or more characteristics, as discussed above. Forexample, such reference data may be categorized based on populationcharacteristics, such as race, gender, or age of the people from whichthe data was collected. Standardized data permits average stomachcharacteristics to be calculated for healthy subjects and subjects withdifferent severity levels of gastric cancer.

The exemplary diagnostic processor system 10 shown in FIG. 1 may be usedto facilitate diagnosis of the severity (e.g., stage) of gastric cancerof the subject. For example, computer-readable instructions foranalyzing and/or processing images and data, generating differenceimages, calculating reliability scores, performing or facilitatingdiagnosis of a particular severity level (e.g., stage) of gastric cancermay be stored in the storage device 14, such as the memory. Theprocessor 12 may execute the computer-readable instructions tofacilitate diagnosis of the subject. As an output device 18, a displaymay be configured to display one or more of: the subject image,generated difference images, the reference images, non-image subjectdata, and the diagnosis received from the processor, including thereliability score.

Based on the diagnosis of the severity (e.g., stage) of gastric cancer,the subject may be appropriately treated. The processor may determinewhich treatment is appropriate based on the severity (e.g., stage) ofgastric cancer and output treatment information accordingly. Forexample, some small stage 0 and stage IA cancers may be treated byendoscopic resection. Other stage 0 and stage I gastric cancers may betreated by surgery to remove the tumor(s), such as by subtotal or totalgastrectomy, in which part or all of the stomach is removed, along withnearby lymph nodes. All stages of gastric cancer may additionally oralternatively be treated by chemotherapy or chemoradiation to shrink thecancer. In some cases, a laser beam directed through an endoscope (along, flexible tube passed down the throat) can be used to destroy thetumor and relieve obstruction without surgery. If needed, a stent may beplaced where the esophagus and stomach meet to help keep it open andallow food to pass through it. This can also be done at the junction ofthe stomach and the small intestine. Targeted therapy can also behelpful in treating advanced gastric cancers. For example, Trastuzumabcan be added to chemotherapy for subjects with tumors that areHER2-positive. Ramucirumab may also be used by itself or withchemotherapy. Pembrolizumab, which is an immunotherapy drug, may also beadministered.

Exemplary treatments based on gastric cancer depth and metastasis areshown in Table 6 below.

TABLE 6 Treatment Based on Depth and Metastasis Metastasis to Metastasisto Depth No metastasis lymph node distant place T1 Mucosal EMR Surgery →Chemotherapy (M) layer Chemotherapy T1 Submucosal Surgery → Surgery →(SM) layer Chemotherapy Chemotherapy T2 Muscular Surgery → Surgery →layer Chemotherapy Chemotherapy T3 Over Surgery → Chemotherapy →muscular Chemotherapy Surgery layer T4 Appears Chemotherapy →Chemotherapy → on the Surgery Surgery surface of the stomach

In some instances, the treatment recommendation output by the processormay include instructions to perform Computed Tomography (CT) examinationand/or Magnetic Resonance Imaging (MRI) examination. For example, whenthe diagnosis result (severity) suggests that the cancer may havemetastasized to another organ, the processor may be programmed to outputa treatment recommendation that includes performing a CT and/or MRI toconfirm whether the cancer has metastasized to another organ.

Some embodiments may employ systems and methods for image analyticsusing machine learning. For example, the images, including wavelengthand difference images, the diagnosis (e.g., gastric cancer stage,reliability score), and medical record information (e.g., information ofpatient's age, gender, nationality, medical history, and other tests,such as X-ray imaging examination, CT examination, MRI examination, PETexamination, ultrasound examination, pathological examination results,or the like) may be input into a system, such as the system 10 (FIG. 1).The system may be configured to execute a program to “learn” the imageand the cancer diagnosis, and determine a relationship between the imageand the cancer. For example, the program may be taught to analyze imagesby providing the program with training information that includespreviously-analyzed images and associated diagnoses, as well as otherrelevant clinical, demographic, and external data. The system may learnthe image and the diagnosis (e.g., cancer stage), and determine arelationship between the image and the diagnosis (e.g., cancer stage).The system may also learn the image, the cancer, and the stage, and themedical record information to determine each relationship.

Technical effects of the present disclosure include the accurate andconsistent diagnoses of various gastric conditions and severity levelsthereof, as well as providing decision support tools for user-diagnosisof subjects. For example, technical effects may include thevisualization of subject image and non-image information together in aholistic, intuitive, and uniform manner, facilitating accurate andobjective diagnosis by a user. Additionally, the present systems,methods, and computer-readable media enable the generation of subjectabnormality images and reference abnormality images of known gastricconditions and/or severity levels thereof, and the combination of suchimages with other clinical tests, to facilitate quantitative assessmentand diagnosis of gastric conditions and their severity level. Thedisclosed systems, methods, and computer-readable media enable analysisof multiple parameters, including both image and non-image data, toaccurately and objectively diagnose severity levels of gastricconditions.

It will be appreciated that any of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. Also,various presently unforeseen or unanticipated alternatives,modifications, variations, or improvements therein may be subsequentlymade by those skilled in the art, and are also intended to beencompassed by the following claims.

What is claimed is:
 1. A diagnostic system for determining a severity ofgastric cancer in a subject, comprising: a processor programmed to:obtain subject images of a stomach of a subject collected by anendoscope including a first wavelength image of a first wavelength bandand a second wavelength image of a second wavelength band, generate asubject difference image by subtracting the second wavelength image fromthe first wavelength image, the first wavelength band being longer thanthe second wavelength band, compare the subject difference image withcorresponding reference images representative of different severitylevels of gastric cancer stored in a database, or input the subjectdifference image into a learned model trained using the reference imagesstored in the database to extract a feature pattern corresponding to aseverity level of gastric cancer; and diagnose the subject as having aparticular severity level of gastric cancer.
 2. The diagnostic systemaccording to claim 1, wherein the reference images include: referencewavelength images of different wavelength bands, and referencedifference images obtained from the reference wavelength images.
 3. Thediagnostic system according to claim 1, wherein a plurality of subjectdifference images are generated from different subject wavelength imagesof different wavelength bands.
 4. The diagnostic system according toclaim 1, wherein the processor is further programmed to compare asubject wavelength image with corresponding reference images or inputthe subject wavelength image into the learned model.
 5. The diagnosticsystem according to claim 1, wherein the processor is further programmedto determine a severity level of gastric cancer for each of multiplelayers of a wall of the stomach of the subject by comparing a pluralityof subject images with corresponding reference images.
 6. The diagnosticsystem according to claim 5, wherein the severity level of gastriccancer is determined for each of a mucosal layer, submucosal layer, andmuscle layer of the wall of the stomach of the subject.
 7. Thediagnostic system according to claim 5, wherein the subject is diagnosedas having the particular severity level of gastric cancer based on theseverity level of gastric cancer for each of the multiple layers of thewall of the stomach.
 8. The diagnostic system according to claim 1,wherein the processor is further programmed to determine severity levelsof gastric cancer at various depths a wall of the stomach of the subjectby comparing a plurality of subject images with corresponding referenceimages.
 9. The diagnostic system according to claim 1, wherein theprocessor is further programmed to output treatment recommendation basedon the diagnosis of the severity level of gastric cancer.
 10. Thediagnostic system according to claim 1, wherein the processor is furtherprogrammed to recommend a Computerized Tomography (CT) examination orMagnetic Resonance Imaging (MRI) examination to determine whether cancerhas metastasized to another organ based on the diagnosis of the severitylevel of gastric cancer.
 11. A method for determining a severity ofgastric cancer in a subject, comprising: obtaining subject images of astomach of a subject collected by an endoscope including a firstwavelength image of a first wavelength band and a second wavelengthimage of a second wavelength band, generating, via a processor, asubject difference image by subtracting the second wavelength image fromthe first wavelength image, the first wavelength band being longer thanthe second wavelength band, comparing, via the processor, the subjectdifference image with corresponding reference images representative ofdifferent severity levels of gastric cancer stored in a database, orinputting the subject difference image into a learned model trainedusing the reference images stored in the database to extract a featurepattern corresponding to a severity level of gastric cancer; anddiagnosing, via the processor, the subject as having a particularseverity level of gastric cancer.
 12. The method according to claim 11,wherein the reference images include: reference wavelength images ofdifferent wavelength bands, and reference difference images obtainedfrom the reference wavelength images.
 13. The method according to claim11, wherein a plurality of subject difference images are generated fromdifferent subject wavelength images of different wavelength bands. 14.The method according to claim 11, further comprising comparing a subjectwavelength image with corresponding reference images or inputting thesubject wavelength image into the learned model.
 15. The methodaccording to claim 11, further comprising determining a severity levelof gastric cancer for each of multiple layers of a wall of the stomachof the subject by comparing a plurality of subject images withcorresponding reference images.
 16. The method according to claim 15,wherein the severity level of gastric cancer is determined for each of amucosal layer, submucosal layer, and muscle layer of the wall of thestomach of the subject.
 17. The method according to claim 15, thesubject is diagnosed as having the particular severity level of gastriccancer based on the severity level of gastric cancer for each of themultiple layers of the wall of the stomach.
 18. The method according toclaim 11, further comprising determining severity levels of gastriccancer at various depths a wall of the stomach of the subject bycomparing a plurality of subject images with corresponding referenceimages.
 19. The method according to claim 11, further comprisingtreating the subject based on the diagnosis of the severity of gastriccancer.
 20. A computer-readable storage medium storing acomputer-executable program that causes a computer to perform functionscomprising: obtaining subject images of a stomach of a subject collectedby an endoscope including a first wavelength image of a first wavelengthband and a second wavelength image of a second wavelength band,generating a subject difference image by subtracting the secondwavelength image from the first wavelength image, the first wavelengthband being longer than the second wavelength band, comparing the subjectdifference image with corresponding reference images representative ofdifferent severity levels of gastric cancer stored in a database, orinputting the subject difference image into a learned model trainedusing the reference images stored in the database to extract a featurepattern corresponding to a severity level of gastric cancer; anddiagnosing the subject as having a particular severity level of gastriccancer.